7.8 數據處理 · StuQ《Node.js微信開發》

# 數據處理 ## csv處理 * i-csv * import-csv * export-csv * json2csv * fastcsv ### json2csv json to csv converter [https://github.com/i5ting/json2csv](https://github.com/i5ting/json2csv) ## import-csv [https://github.com/i5ting/import-csv](https://github.com/i5ting/import-csv) 默認使用gbk編碼 ~~~ var import_csv = require('import-csv') import_csv('test.csv',function(err, data){ console.log(data); }) ~~~ 指定字符編碼機 ~~~ var import_csv = require('import-csv') import_csv('test.csv',function(err, data){ console.log(data); }, 'gbk') ~~~ ## export-csv [https://github.com/i5ting/export-csv](https://github.com/i5ting/export-csv) 默認使用gbk編碼 ~~~ var export_csv = require('.') var data = [ {a:1,b:2}, {a:2,b:2}, {a:3,b:2} ] export_csv(data, 'test1.csv') ~~~ 過濾item，重寫數據 ~~~ var export_csv = require('.') var data = [ { a: 1, b: 2 }, { a: 2, b: 2 }, { a: 3, b: 2 } ] export_csv(data, 'test3.csv', function (item) { for (var key in item) { item[key] = 'yy +' + item[key]; } return item; }) ~~~ 指定回調函數 ~~~ var export_csv = require('.') var data = [ {a:1,b:2}, {a:2,b:2}, {a:3,b:2} ] export_csv(data, 'test2.csv', function (item) { return item; }, function () { console.log('end...'); }) ~~~ 指定回調函數，并生成header ~~~ var export_csv = require('.') var data = [ {a:1,b:2}, {a:2,b:2}, {a:3,b:2} ] export_csv(data, 'test2.csv', function (item) { return item; }, function () { console.log('end...'); }, true) ~~~ ## mongodb如何統計 ~~~ var json2csv = require('json2csv'); require('../db.js'); var fs = require('fs'); var Order = require('../app/models/order'); var Activity = require('../app/models/activity'); var Wechat = require('../app/models/wechat'); var Contact = require('../app/models/contact'); function main(){ Order.model.aggregate([ {$match : {status2 : {$exists : true} }}, {$group : {_id : "$activity" , count : {$sum : "$product_count"}}} ]).exec(function (err, data) { var obj = []; for (var i = data.length - 1; i >= 0; i--) { var json = { "url" : "http://shop.mengxiaoban.cn/iscroll.html?id=" + String(data[i]._id), "count" : String(data[i].count) } obj.push(json); console.log(json.url); } var opts = { data: obj, fields: ['url','count'], fieldNames: ['url','count'], quotes: '' }; json2csv(opts, function(err, csv) { if (err) console.log(err); fs.writeFile('./1119todayurl.csv', csv, function(err) { if (err) throw err; console.log('file saved'); }); }); }) } } main(); ~~~ AGGREGATION 關聯《SQL to Aggregation Mapping Chart》 [http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/](http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/) ## 優化查詢速度了解索引優化 ~~~ ContactSchema = new Schema({ ... owner: { type: Schema.ObjectId, required: true, index: true } }); ~~~ 也可以這樣的 ~~~ ContactSchema.ensureIndexes(owner); ~~~ ## 了解explain ~~~ db.usermodels.find({ '_id' :{ "$gt" :ObjectId("55940ae59c39572851075bfd") } }).explain() ~~~ 關注點 * stage：查詢策略 * nReturned：返回的文檔行數 * needTime：耗時（毫秒） * indexBounds：所用的索引 [http://docs.mongodb.org/v3.0/reference/explain-results/](http://docs.mongodb.org/v3.0/reference/explain-results/) ## 了解profile profile級別有三種： * 0：不開啟 * 1：記錄慢命令，默認為大于100ms * 2：記錄所有命令 * 3、查詢profiling記錄開啟 ~~~ db.setProfilingLevel(2, 20) ~~~ 默認記錄在system.profile中 ~~~ db['system.profile'].find() ~~~ ## MONGODB的部署 * replset * shard 我寫的《 mongodb運維之副本集實踐》 [https://cnodejs.org/topic/5590adbbebf9c92d17e734de](https://cnodejs.org/topic/5590adbbebf9c92d17e734de)