Release 2.19.4

feat: need to sync sourceNpmRegistry also (#1153 )
make sure package data be update event replicate down. https://github.com/npm/registry/issues/129
2017-03-26 23:23:41 +08:00 · 2017-03-24 00:18:34 +08:00 · 2017-03-09 10:29:57 +08:00 · 2017-02-22 00:06:28 +08:00 · 2017-02-22 00:05:29 +08:00 · 2017-01-05 19:41:23 +08:00
63 changed files with 1526 additions and 810 deletions
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,9 +2,10 @@ sudo: false
 language: node_js
 node_js:
  - '4'
-  - '5'
+  - '6'
+  - '7'
 addons:
  - postgresql: '9.3'
 script: 'make test-travis-all'
 after_script:
-  - "npm i codecov && codecov"
+  - 'npm i codecov && codecov'
--- a/3
+++ b/3
@@ -6,4 +6,7 @@ alsotang <alsotang@gmail.com> (https://github.com/alsotang)
 4simple <wondger@qq.com> (https://github.com/4simple)
 afc163 <afc163@gmail.com> (https://github.com/afc163)
 Yuwei Ba <i@xiaoba.me> (https://github.com/ibigbug)
+dickeylth <dickeylth@gmail.com> (https://github.com/dickeylth)
+wenbing <wenbing@users.noreply.github.com> (https://github.com/wenbing)
+21paradox <1036339815@qq.com> (https://github.com/21paradox)

--- a/History.md
+++ b/History.md
@@ -1,4 +1,161 @@

+2.19.4 / 2017-03-26
+==================
+
+  * feat: need to sync sourceNpmRegistry also (#1153)
+  * docs: change user.json to utf8mb4
+
+2.19.3 / 2017-02-22
+==================
+
+  * fix: should get package from orginal registry when package is unpublished (#1130)
+
+2.19.2 / 2017-01-05
+==================
+
+  * fix: should auto sync un-deprecate message (#1105)
+
+2.19.1 / 2016-12-29
+==================
+
+  * fix: try to use the best repository url (#1102)
+
+2.19.0 / 2016-12-21
+==================
+
+  * feat: keyword search with limit to support keywords > 100 (#1097)
+
+2.18.0 / 2016-12-05
+==================
+
+  * fix: support downloads total on scope package (#1088)
+  * fix: try to sync from official replicate (#1076)
+  * feat: add change password script (#1070)
+  * test: skip always fail tests
+  * test: add node v7
+  * feat: show more sync info
+
+2.17.2 / 2016-11-13
+==================
+
+  * fix: ignore long package name on unpublished sync (#1067)
+
+2.17.1 / 2016-11-08
+==================
+
+  * fix: add publish_time for private packages (#1061)
+
+2.17.0 / 2016-11-03
+==================
+
+  * feat: make snyk.io url configable (#1058)
+
+2.16.2 / 2016-09-27
+==================
+
+  * fix: try to use config.registryHost first on setDownloadURL (#1044)
+
+2.16.1 / 2016-08-22
+==================
+
+  * refactor: refine publishable's code (#1022)
+
+2.16.0 / 2016-08-22
+==================
+
+  * feat: admin can do everything (#1021)
+
+2.15.0 / 2016-08-22
+==================
+
+  * feat: return dist-tag on package registry ([#1020](https://github.com/cnpm/cnpmjs.org/issues/1020))
+  * chore(package): update supertest to version 2.0.0 ([#1004](https://github.com/cnpm/cnpmjs.org/issues/1004))
+
+2.14.0 / 2016-08-04
+==================
+
+  * feat: password may contains ":" ([#999](https://github.com/cnpm/cnpmjs.org/issues/999))
+  * fix: limit sync fails email notice ([#1006](https://github.com/cnpm/cnpmjs.org/issues/1006))
+
+2.13.0 / 2016-07-26
+==================
+
+  * feat: enable maxrequests middleware ([#1003](https://github.com/cnpm/cnpmjs.org/issues/1003))
+
+2.12.2 / 2016-07-11
+==================
+
+  * fix: getModuleByRange don't list all packages ([#990](https://github.com/cnpm/cnpmjs.org/issues/990))
+  * fix: should show new version package count ([#984](https://github.com/cnpm/cnpmjs.org/issues/984))
+
+2.12.1 / 2016-07-01
+==================
+
+  * fix: make sure chagnes stream destroy ([#982](https://github.com/cnpm/cnpmjs.org/issues/982))
+  * chore(package): update semver to version 5.2.0 ([#978](https://github.com/cnpm/cnpmjs.org/issues/978))
+  * deps: use ^ instead of ~ ([#976](https://github.com/cnpm/cnpmjs.org/issues/976))
+  * chore(package): update mini-logger to version 1.1.1 ([#973](https://github.com/cnpm/cnpmjs.org/issues/973))
+
+2.12.0 / 2016-06-26
+==================
+
+  * fix: logger seperator should be one EOL ([#972](https://github.com/cnpm/cnpmjs.org/issues/972))
+  * feat: add security check badge for public package ([#971](https://github.com/cnpm/cnpmjs.org/issues/971))
+
+2.11.0 / 2016-06-25
+==================
+
+  * feat: add changes stream syncer ([#970](https://github.com/cnpm/cnpmjs.org/issues/970))
+  * chore(package): update pg to version 5.1.0 ([#953](https://github.com/cnpm/cnpmjs.org/issues/953))
+
+2.10.1 / 2016-06-05
+==================
+
+  * fix: should sync missing public scoped package on install ([#946](https://github.com/cnpm/cnpmjs.org/issues/946))
+  * chore(package): update bytes to version 2.4.0 ([#943](https://github.com/cnpm/cnpmjs.org/issues/943))
+  * userService ([#926](https://github.com/cnpm/cnpmjs.org/issues/926))
+  * chore(package): update should to version 8.4.0 ([#928](https://github.com/cnpm/cnpmjs.org/issues/928))
+  * chore(package): update humanize-ms to version 1.2.0 ([#927](https://github.com/cnpm/cnpmjs.org/issues/927))
+  * chore(package): update kcors to version 1.2.1 ([#918](https://github.com/cnpm/cnpmjs.org/issues/918))
+  * chore(package): update urllib to version 2.9.0 ([#898](https://github.com/cnpm/cnpmjs.org/issues/898))
+
+2.10.0 / 2016-04-15
+==================
+
+  * feat: show tarball url on package page ([#894](https://github.com/cnpm/cnpmjs.org/issues/894))
+  * chore(package): update koa-mock to version 1.6.1 ([#891](https://github.com/cnpm/cnpmjs.org/issues/891))
+
+2.9.5 / 2016-04-12
+==================
+
+  * fix: change logo url to a better https source
+  * fix: http://cnpmjs.org/package/fms pre style ([#739](https://github.com/cnpm/cnpmjs.org/issues/739))
+
+2.9.4 / 2016-04-09
+==================
+
+  * fix: don't sync constructor package on exists mode ([#883](https://github.com/cnpm/cnpmjs.org/issues/883))
+  * Update utility to version 1.7.0 🚀
+  * chore: update sponsor link
+
+2.9.3 / 2016-04-05
+==================
+
+  * fix: use better diff time to check sync status
+  * Update sequelize to version 3.21.0 🚀
+  * chore(package): update agentkeepalive to version 2.1.0
+  * chore(package): update pg to version 4.5.2
+
+2.9.2 / 2016-03-29
+==================
+
+  * fix: override antd for ul & ol list number & icon.
+
+2.9.1 / 2016-03-29
+==================
+
+  * refactor: add more ua info on syncer
+
 2.9.0 / 2016-03-26
 ==================

--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -1,6 +1,6 @@
 This software is licensed under the MIT License.

-Copyright(c) cnpmjs.org and other contributors.
+Copyright(c) cnpm and other contributors.

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
--- a/29
+++ b/29
@@ -4,15 +4,8 @@ TIMEOUT = 30000
 MOCHA_OPTS =
 DB = sqlite

-install:
-	@npm install --build-from-source --registry=https://registry.npm.taobao.org \
-		--disturl=https://npm.taobao.org/mirrors/node
-
-install-production production:
-	@NODE_ENV=production $(MAKE) install
-
-jshint: install
-	@-node_modules/.bin/jshint ./
+jshint:
+	@node_modules/.bin/jshint .

 init-database:
 	@node test/init_db.js
@@ -25,12 +18,11 @@ init-pg:
 	@psql -c 'DROP DATABASE IF EXISTS cnpmjs_test;'
 	@psql -c 'CREATE DATABASE cnpmjs_test;'

-test: install init-database
+test: init-database
 	@NODE_ENV=test DB=${DB} node_modules/.bin/mocha \
 		--reporter $(REPORTER) \
 		--timeout $(TIMEOUT) \
 		--require should \
-		--require should-http \
 		--require thunk-mocha \
 		--require ./test/init.js \
 		$(MOCHA_OPTS) \
@@ -47,7 +39,7 @@ test-pg: init-pg

 test-all: test-sqlite test-mysql

-test-cov cov: install init-database
+test-cov cov: init-database
 	@NODE_ENV=test DB=${DB} node \
 		node_modules/.bin/istanbul cover \
 		node_modules/.bin/_mocha \
@@ -55,7 +47,6 @@ test-cov cov: install init-database
 		--reporter $(REPORTER) \
 		--timeout $(TIMEOUT) \
 		--require should \
-		--require should-http \
 		--require thunk-mocha \
 		--require ./test/init.js \
 		$(MOCHA_OPTS) \
@@ -67,7 +58,7 @@ test-cov-sqlite:
 test-cov-mysql: init-mysql
 	@$(MAKE) test-cov DB=mysql

-test-travis: install init-database
+test-travis: init-database
 	@NODE_ENV=test DB=${DB} CNPM_SOURCE_NPM=https://registry.npmjs.com CNPM_SOURCE_NPM_ISCNPM=false \
 		node \
 		node_modules/.bin/istanbul cover \
@@ -76,7 +67,6 @@ test-travis: install init-database
 		--reporter dot \
 		--timeout $(TIMEOUT) \
 		--require should \
-		--require should-http \
 		--require thunk-mocha \
 		--require ./test/init.js \
 		$(MOCHA_OPTS) \
@@ -93,20 +83,19 @@ test-travis-pg:
 	@psql -c 'CREATE DATABASE cnpmjs_test;' -U postgres
 	@DB_PORT=5432 DB_USER=postgres $(MAKE) test-travis DB=postgres

-test-travis-all: test-travis-sqlite test-travis-mysql test-travis-pg
+test-travis-all: jshint test-travis-sqlite test-travis-mysql test-travis-pg

 dev:
 	@NODE_ENV=development node node_modules/.bin/node-dev dispatch.js

-contributors: install
+contributors:
 	@node_modules/.bin/contributors -f plain -o AUTHORS

-autod: install
+autod:
 	@node_modules/.bin/autod -w \
 		--prefix "~" \
 		--exclude public,view,docs,backup,coverage \
 		--dep mysql \
-		--keep should,supertest,should-http,chunkstream,mm,pedding
-	@$(MAKE) install
+		--keep should,supertest,chunkstream,mm,pedding

 .PHONY: test
--- a/README.md
+++ b/README.md
@@ -3,18 +3,21 @@ cnpmjs.org

 [![NPM version][npm-image]][npm-url]
 [![build status][travis-image]][travis-url]
-[![Test coverage][cov-image]][cov-url]
+[![Test coverage][codecov-image]][codecov-url]
 [![David deps][david-image]][david-url]
+[![Known Vulnerabilities][snyk-image]][snyk-url]
 [![npm download][download-image]][download-url]

 [npm-image]: http://cnpmjs.org/badge/v/cnpmjs.org.svg?style=flat-square
 [npm-url]: http://cnpmjs.org/package/cnpmjs.org
 [travis-image]: https://img.shields.io/travis/cnpm/cnpmjs.org.svg?style=flat-square
 [travis-url]: https://travis-ci.org/cnpm/cnpmjs.org
-[cov-image]: http://codecov.io/github/cnpm/cnpmjs.org/coverage.svg?branch=master
-[cov-url]: http://codecov.io/github/cnpm/cnpmjs.org?branch=master
+[codecov-image]: https://codecov.io/gh/cnpm/cnpmjs.org/branch/master/graph/badge.svg
+[codecov-url]: https://codecov.io/gh/cnpm/cnpmjs.org
 [david-image]: https://img.shields.io/david/cnpm/cnpmjs.org.svg?style=flat-square
 [david-url]: https://david-dm.org/cnpm/cnpmjs.org
+[snyk-image]: https://snyk.io/test/npm/cnpmjs.org/badge.svg?style=flat-square
+[snyk-url]: https://snyk.io/test/npm/cnpmjs.org
 [download-image]: https://img.shields.io/npm/dm/cnpmjs.org.svg?style=flat-square
 [download-url]: https://npmjs.org/package/cnpmjs.org

@@ -111,7 +114,7 @@ Tips: make sure your code is following the [node-style-guide](https://github.com

 ## Sponsors

- [![阿里云](https://static.aliyun.com/images/www-summerwind/logo.gif)](https://www.aliyun.com?from=cnpmjs.org) (2016.2 - now)
+- [![阿里云](https://static.aliyun.com/images/www-summerwind/logo.gif)](http://click.aliyun.com/m/4288/) (2016.2 - now)
 - [![UCloud云计算](https://www.ucloud.cn/static/style/images/about/logo.png)](http://www.ucloud.cn?sem=sdk-CNPMJS) (2015.3 - 2016.3)

 ## License
--- a/bin/change_password.js
+++ b/bin/change_password.js
@@ -0,0 +1,24 @@
+"use strict";
+// Only support for ./services/DefaultUserService. If you use custom user service, ignore this file.
+// call with:
+// $ node ./bin/change_password.js 'username' 'new_password'
+
+var UserModel = require('../models').User;
+var co = require('co');
+var utility = require('utility');
+
+var username = process.argv[2];
+var newPassword = process.argv[3];
+
+co(function * () {
+  var user = yield UserModel.find({where: {name: username}});
+  var salt = user.salt;
+  console.log(`user original password_sha: ${user.password_sha}`);
+  var newPasswordSha = utility.sha1(newPassword + salt);
+  user.password_sha = newPasswordSha;
+  user = yield user.save();
+  console.log(`change user password successful!! user new password_sha: ${user.password_sha}`);
+  process.exit(0);
+}).catch(function (e) {
+  console.log(e);
+});
--- a/common/logger.js
+++ b/common/logger.js
@@ -1,62 +1,56 @@
-/**!
- * cnpmjs.org - common/logger.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
+const debug = require('debug')('cnpmjs.org:logger');
+const formater = require('error-formater');
+const Logger = require('mini-logger');
+const utility = require('utility');
+const util = require('util');
+const os = require('os');
+const config = require('../config');
+const mail = require('./mail');

-var formater = require('error-formater');
-var Logger = require('mini-logger');
-var utility = require('utility');
-var util = require('util');
-var config = require('../config');
-var mail = require('./mail');
+const isTEST = process.env.NODE_ENV === 'test';
+const categories = ['sync_info', 'sync_error'];

-var isTEST = process.env.NODE_ENV === 'test';
-var categories = ['sync_info', 'sync_error'];
-
-var logger = module.exports = Logger({
+const logger = module.exports = Logger({
  categories: categories,
  dir: config.logdir,
  duration: '1d',
  format: '[{category}.]YYYY-MM-DD[.log]',
  stdout: config.debug && !isTEST,
-  errorFormater: errorFormater
+  errorFormater: errorFormater,
+  seperator: os.EOL,
 });

-var to = [];
+const to = [];
 for (var user in config.admins) {
  to.push(config.admins[user]);
 }

 function errorFormater(err) {
-  var msg = formater.both(err);
+  const msg = formater.both(err);
  mail.error(to, msg.json.name, msg.text);
  return msg.text;
 }

 logger.syncInfo = function () {
-  var args = [].slice.call(arguments);
+  const args = [].slice.call(arguments);
  if (typeof args[0] === 'string') {
    args[0] = util.format('[%s][%s] ', utility.logDate(), process.pid) + args[0];
  }
+  if (debug.enabled) {
+    debug.apply(debug, args);
+  }
  logger.sync_info.apply(logger, args);
 };

 logger.syncError =function () {
-  var args = [].slice.call(arguments);
+  const args = [].slice.call(arguments);
  if (typeof args[0] === 'string') {
    args[0] = util.format('[%s][%s] ', utility.logDate(), process.pid) + args[0];
  }
+  if (debug.enabled) {
+    debug.apply(debug, args);
+  }
  logger.sync_error.apply(logger, arguments);
 };
--- a/config/index.js
+++ b/config/index.js
@@ -1,18 +1,5 @@
-/**
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <m@fengmk2.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var mkdirp = require('mkdirp');
 var copy = require('copy-to');
 var path = require('path');
@@ -26,6 +13,7 @@ var dataDir = path.join(process.env.HOME || root, '.cnpmjs.org');

 var config = {
  version: version,
+  dataDir: dataDir,

  /**
   * Cluster mode
@@ -93,7 +81,7 @@ var config = {
    }
  },

-  logoURL: '//ww4.sinaimg.cn/large/69c1d4acgw1ebfly5kjlij208202oglr.jpg', // cnpm logo image url
+  logoURL: 'https://os.alipayobjects.com/rmsportal/oygxuIUkkrRccUz.jpg', // cnpm logo image url
  adBanner: '',
  customReadmeFile: '', // you can use your custom readme file instead the cnpm one
  customFooter: '', // you can add copyright and site total script html here
@@ -175,6 +163,7 @@ var config = {
  // but sometimes will request it for some package infomations
  // please don't change it if not necessary
  officialNpmRegistry: 'https://registry.npmjs.com',
+  officialNpmReplicate: 'https://replicate.npmjs.com',

  // sync source, upstream registry
  // If you want to directly sync from official npm's registry
@@ -209,11 +198,17 @@ var config = {
  // sync devDependencies or not, default is false
  syncDevDependencies: false,

+  // changes streaming sync
+  syncChangesStream: false,
+  handleSyncRegistry: 'http://127.0.0.1:7001',
+
  // badge subject on http://shields.io/
  badgePrefixURL: 'https://img.shields.io/badge',
  badgeSubject: 'cnpm',

  // custom user service, @see https://github.com/cnpm/cnpmjs.org/wiki/Use-Your-Own-User-Authorization
+  // when you not intend to ingegrate with your company's user system, then use null, it would
+  // use the default cnpm user system
  userService: null,

  // always-auth https://docs.npmjs.com/misc/config#always-auth
@@ -223,6 +218,9 @@ var config = {
  // if you're behind firewall, need to request through http proxy, please set this
  // e.g.: `httpProxy: 'http://proxy.mycompany.com:8080'`
  httpProxy: null,
+
+  // snyk.io root url
+  snykUrl: 'https://snyk.io',
 };

 if (process.env.NODE_ENV !== 'test') {
--- a/controllers/registry/package/dist_tag.js
+++ b/controllers/registry/package/dist_tag.js
@@ -1,17 +1,5 @@
-/**!
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var packageService = require('../../../services/package');

 function ok() {
--- a/controllers/registry/package/download_total.js
+++ b/controllers/registry/package/download_total.js
@@ -1,25 +1,11 @@
-/*!
- * cnpmjs.org - controllers/registry/package/download_total.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   dead_horse <dead_horse@qq.com> (https://github.com/dead-horse)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var DownloadTotal = require('../../../services/download_total');
 var DATE_REG = /^\d{4}-\d{2}-\d{2}$/;

-module.exports = function* downloadTotal () {
-  var range = this.params.range;
-  var name = this.params.name;
+module.exports = function* downloadTotal() {
+  var range = this.params.range || this.params[0] || '';
+  var name = this.params.name || this.params[1];

  range = range.split(':');
  if (range.length !== 2
--- a/controllers/registry/package/list.js
+++ b/controllers/registry/package/list.js
@@ -1,11 +1,3 @@
-/**
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

 /**
@@ -99,7 +91,7 @@ module.exports = function* list() {
      this.status = 404;
      this.body = {
        error: 'not_found',
-        reason: 'document not found'
+        reason: 'document not found',
      };
      return;
    }
@@ -134,6 +126,8 @@ module.exports = function* list() {
    }
    common.setDownloadURL(pkg, this);
    pkg._cnpm_publish_time = row.publish_time;
+    pkg.publish_time = pkg.publish_time || row.publish_time;
+
    versions[pkg.version] = pkg;

    var t = times[pkg.version] = row.publish_time ? new Date(row.publish_time) : row.gmt_modified;
--- a/controllers/registry/package/list_since.js
+++ b/controllers/registry/package/list_since.js
@@ -1,19 +1,5 @@
-/**!
- * cnpmjs.org - controllers/registry/package/list_since.js
- *
- * Copyright(c) fengmk2 and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var packageService = require('../../../services/package');

 var A_WEEK_MS = 3600000 * 24 * 7;
@@ -21,7 +7,7 @@ var A_WEEK_MS = 3600000 * 24 * 7;
 // GET /-/all/since?stale=update_after&startkey={key}
 // List packages names since startkey
 // https://github.com/npm/npm-registry-client/blob/master/lib/get.js#L89
-module.exports = function* () {
+module.exports = function* listSince() {
  var query = this.query;
  if (query.stale !== 'update_after') {
    this.status = 400;
@@ -49,7 +35,7 @@ module.exports = function* () {
      Date(), query, this.ip);
  }

-  var names = yield* packageService.listPublicModuleNamesSince(startkey);
+  var names = yield packageService.listPublicModuleNamesSince(startkey);
  var result = { _updated: updated };
  names.forEach(function (name) {
    result[name] = true;
--- a/controllers/registry/package/show.js
+++ b/controllers/registry/package/show.js
@@ -1,17 +1,5 @@
-/**!
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:controllers:registry:package:show');
 var semver = require('semver');
 var packageService = require('../../../services/package');
@@ -47,10 +35,23 @@ module.exports = function* show() {
  if (mod) {
    setDownloadURL(mod.package, this);
    mod.package._cnpm_publish_time = mod.publish_time;
-    var maintainers = yield* packageService.listMaintainers(name);
+    mod.package.publish_time = mod.package.publish_time || mod.publish_time;
+    var rs = yield [
+      packageService.listMaintainers(name),
+      packageService.listModuleTags(name),
+    ];
+    var maintainers = rs[0];
    if (maintainers.length > 0) {
      mod.package.maintainers = maintainers;
    }
+    var tags = rs[1];
+    var distTags = {};
+    for (var i = 0; i < tags.length; i++) {
+      var t = tags[i];
+      distTags[t.tag] = t.version;
+    }
+    // show tags for npminstall faster download
+    mod.package['dist-tags'] = distTags;
    this.jsonp = mod.package;
    return;
  }
--- a/controllers/sync_module_worker.js
+++ b/controllers/sync_module_worker.js
@@ -1,18 +1,5 @@
-/**!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:sync_module_worker');
 var co = require('co');
 var gather = require('co-gather');
@@ -38,7 +25,11 @@ var User = require('../models').User;
 var os = require('os');

 var USER_AGENT = 'sync.cnpmjs.org/' + config.version +
-  ' hostname/' + os.hostname() + ' ' + urllib.USER_AGENT;
+  ' hostname/' + os.hostname() +
+  ' syncModel/' + config.syncModel +
+  ' syncInterval/' + config.syncInterval +
+  ' syncConcurrency/' + config.syncConcurrency +
+  ' ' + urllib.USER_AGENT;

 function SyncModuleWorker(options) {
  EventEmitter.call(this);
@@ -68,6 +59,7 @@ function SyncModuleWorker(options) {

  this.successes = [];
  this.fails = [];
+  this.updates = [];
 }

 util.inherits(SyncModuleWorker, EventEmitter);
@@ -136,7 +128,7 @@ SyncModuleWorker.prototype.start = function () {
    // sync upstream
    if (that.syncUpstreamFirst) {
      try {
-        yield* that.syncUpstream(that.startName);
+        yield that.syncUpstream(that.startName);
      } catch (err) {
        logger.error(err);
      }
@@ -189,14 +181,16 @@ SyncModuleWorker.prototype._doneOne = function* (concurrencyId, name, success) {
  var that = this;
  // relase the stack: https://github.com/cnpm/cnpmjs.org/issues/328
  defer.setImmediate(function* () {
-    yield* that.next(concurrencyId);
+    yield that.next(concurrencyId);
  });
 };

 SyncModuleWorker.prototype.syncUpstream = function* (name) {
  if (config.sourceNpmRegistry.indexOf('registry.npmjs.org') >= 0 ||
-      config.sourceNpmRegistry.indexOf('registry.npmjs.com') >= 0) {
-    this.log('----------------- upstream is npm registry: %s, ignore it -------------------', config.sourceNpmRegistry);
+      config.sourceNpmRegistry.indexOf('registry.npmjs.com') >= 0 ||
+      config.sourceNpmRegistry.indexOf('replicate.npmjs.com') >= 0) {
+    this.log('----------------- upstream is npm registry: %s, ignore it -------------------',
+      config.sourceNpmRegistry);
    return;
  }
  var syncname = name;
@@ -298,6 +292,19 @@ SyncModuleWorker.prototype.next = function* (concurrencyId) {
    return setImmediate(this.finish.bind(this));
  }

+  // try to sync from official replicate when source npm registry is not cnpmjs.org
+  const registry = config.sourceNpmRegistryIsCNpm ? config.sourceNpmRegistry : config.officialNpmReplicate;
+
+  const versions = yield this.syncByName(concurrencyId, name, registry);
+  if (versions && versions.length === 0 && registry === config.officialNpmReplicate) {
+    // need to sync sourceNpmRegistry also
+    // make sure package data be update event replicate down.
+    // https://github.com/npm/registry/issues/129
+    yield this.syncByName(concurrencyId, name, config.officialNpmRegistry);
+  }
+};
+
+SyncModuleWorker.prototype.syncByName = function* (concurrencyId, name, registry) {
  var that = this;
  that.syncingNames[name] = true;
  var pkg = null;
@@ -309,72 +316,109 @@ SyncModuleWorker.prototype.next = function* (concurrencyId) {
  if (common.isPrivateScopedPackage(name)) {
    this.log('[c#%d] [%s] ignore sync private scoped %j package',
      concurrencyId, name, config.scopes);
-    yield* this._doneOne(concurrencyId, name, true);
+    yield this._doneOne(concurrencyId, name, true);
    return;
  }

  // get from npm
+  const packageUrl = '/' + name.replace('/', '%2f');
  try {
-    var result = yield* npmSerivce.request('/' + name.replace('/', '%2f'));
+    var result = yield npmSerivce.request(packageUrl, { registry: registry });
    pkg = result.data;
    status = result.status;
  } catch (err) {
    // if 404
    if (!err.res || err.res.statusCode !== 404) {
      var errMessage = err.name + ': ' + err.message;
-      that.log('[c#%s] [error] [%s] get package error: %s, status: %s',
-        concurrencyId, name, errMessage, status);
-      yield *that._doneOne(concurrencyId, name, false);
+      that.log('[c#%s] [error] [%s] get package(%s%s) error: %s, status: %s',
+        concurrencyId, name, registry, packageUrl, errMessage, status);
+      yield that._doneOne(concurrencyId, name, false);
      return;
    }
  }

+  if (status === 404 && pkg && pkg.reason === 'deleted' && registry === config.officialNpmReplicate) {
+    // unpublished package on replicate.npmjs.com
+    // 404 { error: 'not_found', reason: 'deleted' }
+    // try to read from registry.npmjs.com and get the whole unpublished info
+    try {
+      var result = yield npmSerivce.request(packageUrl, { registry: config.sourceNpmRegistry });
+      pkg = result.data;
+      status = result.status;
+    } catch (err) {
+      // if 404
+      if (!err.res || err.res.statusCode !== 404) {
+        var errMessage = err.name + ': ' + err.message;
+        that.log('[c#%s] [error] [%s] get package(%s%s) error: %s, status: %s',
+          concurrencyId, name, config.sourceNpmRegistry, packageUrl, errMessage, status);
+        yield that._doneOne(concurrencyId, name, false);
+        return;
+      }
+    }
+  }
+
  var unpublishedInfo = null;
  if (status === 404) {
    // check if it's unpublished
-    if (pkg && pkg.time && pkg.time.unpublished && pkg.time.unpublished.time) {
+    // ignore too long package name, see https://github.com/cnpm/cnpmjs.org/issues/1066
+    if (name.length < 80 && pkg && pkg.time && pkg.time.unpublished && pkg.time.unpublished.time) {
      unpublishedInfo = pkg.time.unpublished;
    } else {
      pkg = null;
    }
+  } else {
+    // unpublished package status become to 200
+    if (name.length < 80 && pkg && pkg.time && pkg.time.unpublished && pkg.time.unpublished.time) {
+      unpublishedInfo = pkg.time.unpublished;
+    }
  }

  if (!pkg) {
-    that.log('[c#%s] [error] [%s] get package error: package not exists, status: %s',
-      concurrencyId, name, status);
-    yield* that._doneOne(concurrencyId, name, true);
+    that.log('[c#%s] [error] [%s] get package(%s%s) error: package not exists, status: %s',
+      concurrencyId, name, registry, packageUrl, status);
+    yield that._doneOne(concurrencyId, name, true);
    return;
  }

-  that.log('[c#%d] [%s] pkg status: %d, start...', concurrencyId, name, status);
+  that.log('[c#%d] [%s] package(%s%s) status: %s, dist-tags: %j, time.modified: %s, unpublished: %j, start...',
+    concurrencyId, name, registry, packageUrl, status,
+    pkg['dist-tags'], pkg.time && pkg.time.modified,
+    unpublishedInfo);

  if (unpublishedInfo) {
    try {
-      yield* that._unpublished(name, unpublishedInfo);
+      yield that._unpublished(name, unpublishedInfo);
    } catch (err) {
      that.log('[c#%s] [error] [%s] sync error: %s', concurrencyId, name, err.stack);
-      yield* that._doneOne(concurrencyId, name, false);
+      yield that._doneOne(concurrencyId, name, false);
      return;
    }
-    return yield* that._doneOne(concurrencyId, name, true);
+    return yield that._doneOne(concurrencyId, name, true);
  }

  var versions;
  try {
-    versions = yield* that._sync(name, pkg);
+    versions = yield that._sync(name, pkg);
  } catch (err) {
    that.log('[c#%s] [error] [%s] sync error: %s', concurrencyId, name, err.stack);
-    yield* that._doneOne(concurrencyId, name, false);
+    yield that._doneOne(concurrencyId, name, false);
    return;
  }

+  // has new version
+  if (versions.length > 0) {
+    that.updates.push(name);
+  }
+
  this.log('[c#%d] [%s] synced success, %d versions: %s',
    concurrencyId, name, versions.length, versions.join(', '));
-  yield* this._doneOne(concurrencyId, name, true);
+  yield this._doneOne(concurrencyId, name, true);
+
+  return versions;
 };

 function* _listStarUsers(modName) {
-  var users = yield* packageService.listStarUserNames(modName);
+  var users = yield packageService.listStarUserNames(modName);
  var userMap = {};
  users.forEach(function (user) {
    userMap[user] = true;
@@ -620,6 +664,13 @@ SyncModuleWorker.prototype._sync = function* (name, pkg) {
            deprecated: version.deprecated
          });
        }
+        if (exists.package.deprecated && !version.deprecated) {
+          // remove deprecated info
+          missingDeprecateds.push({
+            id: exists.id,
+            deprecated: undefined,
+          });
+        }
        continue;
      }
    }
@@ -672,7 +723,7 @@ SyncModuleWorker.prototype._sync = function* (name, pkg) {
      continue;
    }
    try {
-      yield* that._syncOneVersion(index, syncModule);
+      yield that._syncOneVersion(index, syncModule);
      syncedVersionNames.push(syncModule.version);
    } catch (err) {
      that.log('    [%s:%d] sync error, version: %s, %s: %s',
@@ -680,7 +731,6 @@ SyncModuleWorker.prototype._sync = function* (name, pkg) {
    }
  }

-
  if (deletedVersionNames.length === 0) {
    that.log('  [%s] no versions need to deleted', name);
  } else {
@@ -786,7 +836,7 @@ SyncModuleWorker.prototype._sync = function* (name, pkg) {
      if (r.error) {
        that.log('    save error, id: %s, error: %s', item.id, r.error.message);
      } else {
-        that.log('    saved, id: %s', item.id);
+        that.log('    saved, id: %s, deprecated: %j', item.id, item.deprecated);
      }
    }
  }
@@ -882,7 +932,10 @@ SyncModuleWorker.prototype._sync = function* (name, pkg) {
 };

 SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePackage) {
-  logger.syncInfo('[sync_module_worker] start sync %s@%s', sourcePackage.name, sourcePackage.version);
+  var delay = Date.now() - sourcePackage.publish_time;
+  logger.syncInfo('[sync_module_worker] delay: %s ms, publish_time: %s, start sync %s@%s',
+    delay, utility.logDate(new Date(sourcePackage.publish_time)),
+    sourcePackage.name, sourcePackage.version);
  var that = this;
  var username = this.username;
  var downurl = sourcePackage.dist.tarball;
@@ -906,9 +959,11 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
    devDependencies = Object.keys(sourcePackage.devDependencies || {});
  }

-  that.log('    [%s:%d] syncing, version: %s, dist: %j, no deps: %s, ' +
+  that.log('    [%s:%d] syncing, delay: %s ms, version: %s, dist: %j, no deps: %s, ' +
   'publish on cnpm: %s, dependencies: %d, devDependencies: %d, syncDevDependencies: %s',
-    sourcePackage.name, versionIndex, sourcePackage.version,
+    sourcePackage.name, versionIndex,
+    delay,
+    sourcePackage.version,
    sourcePackage.dist, that.noDep, that._publish,
    dependencies.length,
    devDependencies.length, this.syncDevDependencies);
@@ -932,7 +987,7 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
  }

  // add module dependence
-  yield* packageService.addDependencies(sourcePackage.name, dependencies);
+  yield packageService.addDependencies(sourcePackage.name, dependencies);

  var shasum = crypto.createHash('sha1');
  var dataSize = 0;
@@ -940,7 +995,14 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
  try {
    // get tarball
    logger.syncInfo('[sync_module_worker] downloading %j to %j', downurl, filepath);
-    var r = yield urllib.request(downurl, options);
+    var r;
+    try {
+      r = yield urllib.request(downurl, options);
+    } catch (err) {
+      logger.syncInfo('[sync_module_worker] download %j to %j error: %s', downurl, filepath, err);
+      throw err;
+    }
+
    var statusCode = r.status || -1;
    // https://github.com/cnpm/cnpmjs.org/issues/325
    // if (statusCode === 404) {
@@ -954,6 +1016,7 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
      var err = new Error('Download ' + downurl + ' fail, status: ' + statusCode);
      err.name = 'DownloadTarballError';
      err.data = sourcePackage;
+      logger.syncInfo('[sync_module_worker] %s', err.message);
      throw err;
    }

@@ -970,6 +1033,7 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
      var err = new Error('Download ' + downurl + ' file size is zero');
      err.name = 'DownloadTarballSizeZeroError';
      err.data = sourcePackage;
+      logger.syncInfo('[sync_module_worker] %s', err.message);
      throw err;
    }

@@ -980,6 +1044,7 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
        ' not match ' + sourcePackage.dist.shasum);
      err.name = 'DownloadTarballShasumError';
      err.data = sourcePackage;
+      logger.syncInfo('[sync_module_worker] %s', err.message);
      throw err;
    }

@@ -998,8 +1063,9 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
      throw err;
    }
    logger.syncInfo('[sync_module_worker] uploaded, saving %j to database', result);
-    var r = yield *afterUpload(result);
-    logger.syncInfo('[sync_module_worker] sync %s@%s done!', sourcePackage.name, sourcePackage.version);
+    var r = yield afterUpload(result);
+    logger.syncInfo('[sync_module_worker] sync %s@%s done!',
+      sourcePackage.name, sourcePackage.version);
    return r;
  } finally {
    // remove tmp file whatever
@@ -1043,7 +1109,7 @@ SyncModuleWorker.prototype._syncOneVersion = function *(versionIndex, sourcePack
    }

    mod.package.dist = dist;
-    var r = yield* packageService.saveModule(mod);
+    var r = yield packageService.saveModule(mod);

    that.log('    [%s:%s] done, insertId: %s, author: %s, version: %s, '
      + 'size: %d, publish_time: %j, publish on cnpm: %s',
--- a/controllers/total.js
+++ b/controllers/total.js
@@ -1,18 +1,5 @@
-/**!
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 const Total = require('../services/total');
 const version = require('../package.json').version;
 const config = require('../config');
--- a/controllers/web/package/search.js
+++ b/controllers/web/package/search.js
@@ -21,8 +21,16 @@ var packageService = require('../../../services/package');
 module.exports = function* search() {
  var params = this.params;
  var word = params.word || params[0];
+  var limit = Number(this.query.limit) || 100;
+
+  if (limit > 10000) {
+    limit = 10000;
+  }
+
  debug('search %j', word);
-  var result = yield* packageService.search(word);
+  var result = yield* packageService.search(word, {
+    limit: limit
+  });

  var match = null;
  for (var i = 0; i < result.searchMatchs.length; i++) {
--- a/controllers/web/package/show.js
+++ b/controllers/web/package/show.js
@@ -1,18 +1,5 @@
-/**!
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:controllers:web:package:show');
 var bytes = require('bytes');
 var giturl = require('giturl');
@@ -123,7 +110,7 @@ module.exports = function* show(next) {
    pkg.repository = null;
  }
  if (pkg.repository && pkg.repository.url) {
-    pkg.repository.weburl = giturl.parse(pkg.repository.url) || pkg.repository.url;
+    pkg.repository.weburl = /^https?:\/\//.test(pkg.repository.url) ? pkg.repository.url : (giturl.parse(pkg.repository.url) || pkg.repository.url);
  }
  if (!pkg.bugs) {
    pkg.bugs = {};
@@ -183,6 +170,17 @@ module.exports = function* show(next) {
    }
  }

+  if (pkg._publish_on_cnpm) {
+    pkg.isPrivate = true;
+  } else {
+    pkg.isPrivate = false;
+    // add security check badge
+    pkg.snyk = {
+      badge: `${config.snykUrl}/test/npm/${pkg.name}/badge.svg?style=flat-square`,
+      url: `${config.snykUrl}/test/npm/${pkg.name}`,
+    };
+  }
+
  yield this.render('package', {
    title: 'Package - ' + pkg.name,
    package: pkg,
--- a/dispatch.js
+++ b/dispatch.js
@@ -1,18 +1,5 @@
-/**!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var childProcess = require('child_process');
 var path = require('path');
 var util = require('util');
--- a/docs/db.sql
+++ b/docs/db.sql
@@ -18,6 +18,7 @@ CREATE TABLE IF NOT EXISTS `user` (
 -- ALTER TABLE `user`
 --   ADD `json` longtext CHARACTER SET utf8 COLLATE utf8_general_ci COMMENT 'json details',
 --   ADD `npm_user` tinyint(1) DEFAULT '0' COMMENT 'user sync from npm or not, 1: true, other: false';
+-- ALTER TABLE `user` CHANGE `json` `json` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci COMMENT 'json details';

 CREATE TABLE IF NOT EXISTS `module_keyword` (
 `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'primary key',
--- a/docs/network.png
+++ b/docs/network.png
--- a/docs/network.puml
+++ b/docs/network.puml
@@ -0,0 +1,98 @@
+@startuml
+
+title cnpmjs.org, npm.taobao.org Network
+
+node "China User" {
+  [cnpm cli]
+}
+
+node "OSS: aliyuncs.com" {
+  [tnpm-hz.oss-cn-hangzhou]
+}
+
+node "SLB: 114.55.80.225 Hangzhou" {
+  [npm.taobao.org]
+  [registry.npm.taobao.org]
+}
+
+node "npmtaobao3: 121.41.*" {
+  [nginx 5]
+  [registry 5]
+  [web 5]
+  [syncer 5]
+}
+
+node "npmtaobao4: 120.26.*" {
+  [nginx 6]
+  [registry 6]
+  [web 6]
+}
+
+node "cnpm6: 47.88.189.193 Singapore" {
+  [cnpmjs.org]
+  [r.cnpmjs.org]
+  [nginx 7]
+  [registry 7]
+  [web 7]
+  [syncer 7]
+}
+
+node "Aliyun CDN" {
+  [cdn.npm.taobao.org]
+}
+
+node "qiniu CDN" {
+  [cnpmjs.up.qiniu.com]
+  [dn-cnpm.qbox.me]
+}
+
+database "taobaonpm DB" {
+  [rds2860*.mysql.rds.aliyuncs.com]
+}
+
+database "cnpm DB" {
+  [rdsqiz*.mysql.rds.aliyuncs.com]
+}
+
+node "npmjs.com" {
+  [registry.npmjs.com]
+  [www.npmjs.com]
+}
+
+[cnpm cli] -down-> [registry.npm.taobao.org]: Read, Write
+[cnpm cli] -down-> [cdn.npm.taobao.org]: Read tgz
+[cnpm cli] -down-> [npm.taobao.org]: "Read /mirrors"
+
+[registry.npm.taobao.org] -down-> [nginx 5]
+[nginx 5] -down-> [registry 5]
+[npm.taobao.org] -down-> [nginx 5]
+[nginx 5] -down-> [web 5]
+[registry.npm.taobao.org] -down-> [nginx 6]
+[nginx 6] -down-> [registry 6]
+[npm.taobao.org] -down-> [nginx 6]
+[nginx 6] -down-> [web 6]
+
+[registry 5] -down-> [rds2860*.mysql.rds.aliyuncs.com]: Read, Write
+[web 5] -down-> [rds2860*.mysql.rds.aliyuncs.com]: Read
+[syncer 5] -down-> [rds2860*.mysql.rds.aliyuncs.com]: Read, Write
+[syncer 5] -down-> [tnpm-hz.oss-cn-hangzhou]: Write
+[syncer 5] -> [r.cnpmjs.org]: Read
+[syncer 5] -> [dn-cnpm.qbox.me]: Read tgz
+[registry 6] -down-> [rds2860*.mysql.rds.aliyuncs.com]: Read, Write
+[web 6] -down-> [rds2860*.mysql.rds.aliyuncs.com]: Read
+
+[cnpmjs.org] -down-> [nginx 7]
+[nginx 7] -down-> [registry 7]
+[r.cnpmjs.org] -down-> [nginx 7]
+[nginx 7] -down-> [web 7]
+
+[registry 7] -down-> [rdsqiz*.mysql.rds.aliyuncs.com]: Read, Write
+[web 7] -down-> [rdsqiz*.mysql.rds.aliyuncs.com]: Read
+[syncer 7] -down-> [rdsqiz*.mysql.rds.aliyuncs.com]: Read, Write
+[syncer 7] -down-> [cnpmjs.up.qiniu.com]: Write
+[syncer 7] -down-> [registry.npmjs.com]: Read
+
+[cdn.npm.taobao.org] -down-> [tnpm-hz.oss-cn-hangzhou]: Read
+[dn-cnpm.qbox.me] -down-> [cnpmjs.up.qiniu.com]: Read
+
+@enduml
--- a/docs/registry-api.md
+++ b/docs/registry-api.md
@@ -9,6 +9,8 @@
 * [User](/docs/registry-api.md#user)
 * [Search](/docs/registry-api.md#search)

+[![Run in Postman](https://run.pstmn.io/button.svg)](https://app.getpostman.com/run-collection/f6c8cb46358039bcd689#?env%5BRegistry%5D=W3sia2V5IjoicmVnaXN0cnkiLCJ0eXBlIjoidGV4dCIsInZhbHVlIjoiaHR0cHM6Ly9yZWdpc3RyeS5ucG0udGFvYmFvLm9yZyIsImVuYWJsZWQiOnRydWV9LHsia2V5IjoicGFja2FnZSIsInZhbHVlIjoiY25wbSIsInR5cGUiOiJ0ZXh0IiwiZW5hYmxlZCI6dHJ1ZX1d)
+
 ## Schema

 All API access is over HTTPS or HTTP,
--- a/docs/web/readme.md
+++ b/docs/web/readme.md
@@ -4,10 +4,10 @@ So `cnpm` is meaning: **Company npm**.

 ## Registry

- Our public registry: [r.cnpmjs.org](//r.cnpmjs.org), syncing from [registry.npmjs.org](https://registry.npmjs.org)
+- Our public registry: [r.cnpmjs.org](https://r.cnpmjs.org), syncing from [registry.npmjs.com](https://registry.npmjs.com)
 - [cnpmjs.org](/) version: <span id="app-version"></span>
 - [Node.js](https://nodejs.org) version: <span id="node-version"></span>
- For developers in China, please visit [the China mirror](https://npm.taobao.org). 中国用户请访问[国内镜像站点](https://npm.taobao.org/)。
+- For developers in China, please visit [the China mirror](https://npm.taobao.org). 中国用户请访问[国内镜像站点](https://npm.taobao.org)。

 <div class="ant-table">
 <table class="downloads">
@@ -67,9 +67,9 @@ Default style is `flat-square`.

 Badge URL: `https://cnpmjs.org/badge/v/cnpmjs.org.svg` ![cnpmjs.org-version-badge](//cnpmjs.org/badge/v/cnpmjs.org.svg)

-* `<0.1.0 & >=0.0.0`: ![red-badge](https://dn-img-shields-io.qbox.me/badge/cnpm-0.0.1-red.svg?style=flat-square)
-* `<1.0.0 & >=0.1.0`: ![red-badge](https://dn-img-shields-io.qbox.me/badge/cnpm-0.1.0-green.svg?style=flat-square)
-* `>=1.0.0`: ![red-badge](https://dn-img-shields-io.qbox.me/badge/cnpm-1.0.0-blue.svg?style=flat-square)
+* `<0.1.0 & >=0.0.0`: ![red-badge](https://img.shields.io/badge/cnpm-0.0.1-red.svg?style=flat-square)
+* `<1.0.0 & >=0.1.0`: ![red-badge](https://img.shields.io/badge/cnpm-0.1.0-green.svg?style=flat-square)
+* `>=1.0.0`: ![red-badge](https://img.shields.io/badge/cnpm-1.0.0-blue.svg?style=flat-square)

 ### Downloads

@@ -145,11 +145,11 @@ $ cnpm info cnpm

 Release [History](/history).

-## npm and cnpm relation
+## npmjs.org, cnpmjs.org and npm.taobao.org relation

-![npm&cnpm](https://dn-cnpm.qbox.me/cnpm-npm-relation.png)
+![npm&cnpm](https://cloud.githubusercontent.com/assets/543405/21505401/fd0b6220-cca1-11e6-86ed-599cc81bb03b.png)

 ## Sponsors

- [![阿里云](https://static.aliyun.com/images/www-summerwind/logo.gif)](https://www.aliyun.com?from=cnpmjs.org) (2016.2 - now)
+- [![阿里云](https://static.aliyun.com/images/www-summerwind/logo.gif)](http://click.aliyun.com/m/4288/) (2016.2 - now)
 - [![UCloud云计算](https://www.ucloud.cn/static/style/images/about/logo.png)](http://www.ucloud.cn?sem=sdk-CNPMJS) (2015.3 - 2016.3)
--- a/lib/common.js
+++ b/lib/common.js
@@ -37,7 +37,7 @@ exports.getCDNKey = function (name, filename) {

 exports.setDownloadURL = function (pkg, ctx, host) {
  if (pkg.dist) {
-    host = host || ctx.host;
+    host = host || config.registryHost || ctx.host;
    pkg.dist.tarball = util.format('%s://%s/%s/download/%s-%s.tgz',
      ctx.protocol,
      host, pkg.name, pkg.name, pkg.version);
--- a/middleware/auth.js
+++ b/middleware/auth.js
@@ -34,13 +34,14 @@ module.exports = function () {
      return yield* unauthorized.call(this, next);
    }

-    authorization = new Buffer(authorization, 'base64').toString().split(':');
-    if (authorization.length !== 2) {
-      return yield* unauthorized.call(this, next);
+    authorization = new Buffer(authorization, 'base64').toString();
+    var pos = authorization.indexOf(':');
+    if (pos === -1) {
+       return yield* unauthorized.call(this, next);
    }

-    var username = authorization[0];
-    var password = authorization[1];
+    var username = authorization.slice(0, pos);
+    var password = authorization.slice(pos + 1);

    var row;
    try {
--- a/middleware/proxy_to_npm.js
+++ b/middleware/proxy_to_npm.js
@@ -1,12 +1,3 @@
-/**!
- * cnpmjs.org - middleware/proxy_to_npm.js
- *
- * Copyright(c) Alibaba Group Holding Limited.
- *
- * Authors:
- *   苏千 <suqian.yf@alipay.com> (http://fengmk2.com)
- */
-
 'use strict';

 /**
@@ -33,11 +24,11 @@ module.exports = function (options) {
  }
  return function* proxyToNpm(next) {
    if (config.syncModel !== 'none') {
-      return yield* next;
+      return yield next;
    }
    // only proxy read requests
    if (this.method !== 'GET' && this.method !== 'HEAD') {
-      return yield* next;
+      return yield next;
    }

    var pathname = this.path;
@@ -49,7 +40,7 @@ module.exports = function (options) {
      }
    }
    if (!match) {
-      return yield* next;
+      return yield next;
    }

    var url = redirectUrl + this.url;
--- a/middleware/publishable.js
+++ b/middleware/publishable.js
@@ -19,8 +19,13 @@ var config = require('../config');
 var debug = require('debug')('cnpmjs.org:middlewares/publishable');

 module.exports = function *publishable(next) {
-  // private mode, only admin user can publish
-  if (config.enablePrivate && !this.user.isAdmin) {
+  // admins always can publish and unpublish
+  if (this.user.isAdmin) {
+    return yield next;
+  }
+
+  // private mode, normal user can't publish and unpublish
+  if (config.enablePrivate) {
    this.status = 403;
    this.body = {
      error: 'no_perms',
@@ -29,28 +34,26 @@ module.exports = function *publishable(next) {
    return;
  }

-  // public mode, all user have permission to publish `scoped package`
+  // public mode, normal user have permission to publish `scoped package`
  // and only can publish with scopes in `ctx.user.scopes`, default is `config.scopes`

  var name = this.params.name || this.params[0];

  // check if is private package list in config
  if (config.privatePackages && config.privatePackages.indexOf(name) !== -1) {
-    return yield* next;
+    return yield next;
  }

-  // scope
+  // scoped module
  if (name[0] === '@') {
    if (checkScope(name, this)) {
-      return yield* next;
+      return yield next;
    }
    return;
  }

  // none-scope
-  if (checkNoneScope(name, this)) {
-    return yield* next;
-  }
+  assertNoneScope(name, this);
 };

 /**
@@ -81,12 +84,7 @@ function checkScope(name, ctx) {
 * check if user have permission to publish without scope
 */

-function checkNoneScope(name, ctx) {
-  // admins unpublished everything
-  if (ctx.user.isAdmin && ctx.method === 'DELETE') {
-    return true;
-  }
-
+function assertNoneScope(name, ctx) {
  ctx.status = 403;
  if (ctx.user.scopes.length === 0) {
    ctx.body = {
--- a/middleware/sync_by_install.js
+++ b/middleware/sync_by_install.js
@@ -1,7 +1,5 @@
-/**!
- * cnpmjs.org - middleware/sync_by_install.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
+/**
+ * Copyright(c) cnpm and other contributors.
 * MIT Licensed
 *
 * Authors:
@@ -17,33 +15,36 @@
 var config = require('../config');

 /**
- * this.allowSync  -  allow sync triggle by cnpm install
+ * {Boolean} this.allowSync  -  allow sync triggle by cnpm install
 */

 module.exports = function* syncByInstall(next) {
  this.allowSync = false;
  if (!config.syncByInstall) {
    // only config.enablePrivate should enable sync on install
-    return yield* next;
+    return yield next;
  }
-  // request not by node, consider it request from web, dont sync
+  // request not by node, consider it request from web, don't sync
  var ua = this.get('user-agent');
  if (!ua || ua.indexOf('node') < 0) {
-    return yield* next;
+    return yield next;
  }

-  // if request with `/xxx?write=true`, meaning the read request using for write, dont sync
+  // if request with `/xxx?write=true`, meaning the read request using for write, don't sync
  if (this.query.write) {
-    return yield* next;
+    return yield next;
  }

  var name = this.params.name || this.params[0];

-  // scoped package dont sync
+  // private scoped package don't sync
  if (name && name[0] === '@') {
-    return yield* next;
+    var scope = name.split('/')[0];
+    if (config.scopes.indexOf(scope) >= 0) {
+      return yield next;
+    }
  }

  this.allowSync = true;
-  yield* next;
+  yield next;
 };
--- a/package.json
+++ b/package.json
@@ -1,9 +1,10 @@
 {
  "name": "cnpmjs.org",
-  "version": "2.9.0",
+  "version": "2.19.4",
  "description": "Private npm registry and web for Enterprise, base on MySQL and Simple Store Service",
  "main": "index.js",
  "scripts": {
+    "dev": "DEBUG=cnpm* node dispatch.js",
    "test": "make jshint && make test",
    "start": "./bin/nodejsctl start && cp History.md docs/web/history.md",
    "status": "./bin/nodejsctl status",
@@ -13,46 +14,51 @@
    "cnpmjs.org": "bin/cli.js"
  },
  "dependencies": {
-    "agentkeepalive": "~2.0.3",
-    "bytes": "~2.3.0",
-    "cfork": "~1.4.0",
-    "co": "~4.6.0",
-    "co-defer": "~1.0.0",
-    "co-gather": "~0.0.1",
-    "co-sleep": "~0.0.1",
-    "commander": "~2.9.0",
-    "copy-to": "~2.0.1",
-    "debug": "~2.2.0",
-    "error-formater": "~1.0.3",
-    "fs-cnpm": "~1.2.0",
+    "agentkeepalive": "^2.2.0",
+    "await-event": "^1.0.0",
+    "bytes": "^2.4.0",
+    "cfork": "^1.5.1",
+    "changes-stream": "^1.1.0",
+    "co": "^4.6.0",
+    "co-defer": "^1.0.0",
+    "co-gather": "^0.0.1",
+    "co-sleep": "^0.0.1",
+    "commander": "^2.9.0",
+    "copy-to": "^2.0.1",
+    "debug": "^2.2.0",
+    "error-formater": "^1.0.3",
+    "fs-cnpm": "^1.2.0",
    "giturl": "^1.0.0",
-    "graceful": "~1.0.0",
-    "gravatar": "~1.5.0",
-    "humanize-ms": "~1.0.1",
-    "humanize-number": "~0.0.2",
-    "is-type-of": "~1.0.0",
-    "kcors": "~1.1.0",
-    "koa": "~1.2.0",
-    "koa-limit": "~1.0.2",
-    "koa-markdown": "~2.0.1",
-    "koa-middlewares": "~2.1.0",
-    "koa-mock": "~1.5.0",
-    "koa-safe-jsonp": "~0.3.0",
-    "markdown-it": "~3.0.6",
-    "mime": "~1.3.4",
-    "mini-logger": "~1.1.0",
-    "mkdirp": "~0.5.0",
-    "moment": "~2.12.0",
-    "mysql": "~2.10.2",
-    "nodemailer": "~1.3.0",
-    "semver": "~5.1.0",
-    "sequelize": "~3.20.0",
-    "thunkify-wrap": "~1.0.4",
-    "treekill": "~1.0.0",
+    "graceful": "^1.0.0",
+    "gravatar": "^1.5.0",
+    "humanize-ms": "^1.2.0",
+    "humanize-number": "^0.0.2",
+    "is-type-of": "^1.0.0",
+    "kcors": "^1.2.1",
+    "koa": "^1.2.0",
+    "koa-limit": "^1.0.2",
+    "koa-markdown": "^2.0.1",
+    "koa-maxrequests": "^1.0.0",
+    "koa-middlewares": "^2.1.0",
+    "koa-mock": "^1.6.1",
+    "koa-safe-jsonp": "^0.3.1",
+    "markdown-it": "^3.0.6",
+    "mime": "^1.3.4",
+    "mini-logger": "^1.1.1",
+    "mkdirp": "^0.5.0",
+    "moment": "^2.12.0",
+    "mysql": "^2.10.2",
+    "mz": "^2.4.0",
+    "nodemailer": "^1.3.0",
+    "normalize-registry-metadata": "^1.1.2",
+    "semver": "^5.2.0",
+    "sequelize": "^3.23.4",
+    "thunkify-wrap": "^1.0.4",
+    "treekill": "^1.0.0",
    "tunnel-agent": "^0.4.0",
-    "urllib": "~2.8.0",
-    "utility": "~1.6.0",
-    "xss": "~0.2.10"
+    "urllib": "^2.11.0",
+    "utility": "^1.8.0",
+    "xss": "^0.2.13"
  },
  "devDependencies": {
    "autod": "*",
@@ -64,12 +70,11 @@
    "mocha": "*",
    "node-dev": "*",
    "pedding": "*",
-    "pg": "~4.4.2",
-    "pg-hstore": "~2.3.2",
-    "should": "~7.1.0",
-    "should-http": "*",
+    "pg": "5",
+    "pg-hstore": "2",
+    "should": "8",
    "sqlite3": "*",
-    "supertest": "*",
+    "supertest": "2",
    "thunk-mocha": "1"
  },
  "homepage": "https://github.com/cnpm/cnpmjs.org",
--- a/public/css/github-markdown.css
+++ b/public/css/github-markdown.css
@@ -453,6 +453,7 @@
  font-size: 85%;
  line-height: 1.45;
  background-color: #f7f7f7;
+  color:#333;
  border-radius: 3px;
 }

--- a/public/css/main.css
+++ b/public/css/main.css
@@ -201,6 +201,12 @@ pre code {
  font-size: 28px;
  line-height: 48px;
 }
+
+.pack-ver a.badge-link {
+  font-size: inherit;
+  border-bottom: 0px;
+}
+
 .pack-lastup {
  font-size: 14px;
  display: inline-block;
@@ -292,3 +298,6 @@ pre code {
  height: 30px;
  margin-right: 10px;
 }
+ul, ol{
+  list-style: initial;
+}
--- a/routes/registry.js
+++ b/routes/registry.js
@@ -1,17 +1,4 @@
-/**!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
-"use strict";
-
-/**
- * Module dependencies.
- */
+'use strict';

 var limit = require('../middleware/limit');
 var login = require('../middleware/login');
@@ -118,6 +105,7 @@ function routes(app) {

  // download times
  app.get('/downloads/range/:range/:name', downloadTotal);
+  app.get(/^\/downloads\/range\/([^\/]+)\/(@[\w\-\.]+\/[\w\-\.]+)$/, downloadTotal);
  app.get('/downloads/range/:range', downloadTotal);

  // GET /-/package/:pkg/dependents
--- a/routes/web.js
+++ b/routes/web.js
@@ -1,19 +1,4 @@
-/**!
- * cnpmjs.org - routes/web.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <m@fengmk2.com> (http://fengmk2.com)
- */
-
-"use strict";
-
-/**
- * Module dependencies.
- */
+'use strict';

 var showPackage = require('../controllers/web/package/show');
 var searchPackage = require('../controllers/web/package/search');
--- a/servers/registry.js
+++ b/servers/registry.js
@@ -1,19 +1,4 @@
-/**!
- * cnpmjs.org - servers/registry.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
-"use strict";
-
-/**
- * Module dependencies.
- */
+'use strict';

 var koa = require('koa');
 var app = module.exports = koa();
@@ -28,25 +13,27 @@ var staticCache = require('../middleware/static');
 var notFound = require('../middleware/registry_not_found');
 var cors = require('kcors');
 var proxyToNpm = require('../middleware/proxy_to_npm');
+var maxrequests = require('koa-maxrequests');

+app.use(maxrequests());
 app.use(block());
 middlewares.jsonp(app);
-app.use(middlewares.rt({headerName: 'X-ReadTime'}));
+app.use(middlewares.rt({ headerName: 'X-ReadTime' }));
 app.use(middlewares.rewrite('/favicon.ico', '/favicon.png'));
 app.use(staticCache);

 app.keys = ['todokey', config.sessionSecret];
 app.proxy = true;
-app.use(middlewares.bodyParser({jsonLimit: config.jsonLimit}));
+app.use(middlewares.bodyParser({ jsonLimit: config.jsonLimit }));
 app.use(cors({
-  allowMethods: 'GET,HEAD'
+  allowMethods: 'GET,HEAD',
 }));
 app.use(auth());
 app.use(proxyToNpm());
 app.use(notFound);

 if (config.enableCompress) {
-  app.use(middlewares.compress({threshold: 150}));
+  app.use(middlewares.compress({ threshold: 150 }));
 }
 app.use(middlewares.conditional());
 app.use(middlewares.etag());
--- a/servers/web.js
+++ b/servers/web.js
@@ -1,19 +1,4 @@
-/**!
- * cnpmjs.org - servers/web.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <m@fengmk2.com> (http://fengmk2.com)
- */
-
-"use strict";
-
-/**
- * Module dependencies.
- */
+'use strict';

 var opensearch = require('../middleware/opensearch');
 var notFound = require('../middleware/web_not_found');
@@ -32,6 +17,7 @@ var path = require('path');
 var http = require('http');
 var koa = require('koa');
 var fs = require('fs');
+var maxrequests = require('koa-maxrequests');

 var app = koa();

@@ -39,6 +25,7 @@ jsonp(app);

 var rootdir = path.dirname(__dirname);

+app.use(maxrequests());
 app.use(block());
 app.use(middlewares.rt({headerName: 'X-ReadTime'}));
 app.use(middlewares.rewrite('/favicon.ico', '/favicon.png'));
--- a/services/npm.js
+++ b/services/npm.js
@@ -1,21 +1,7 @@
-/**!
- * cnpmjs.org - services/npm.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- *   dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var ms = require('humanize-ms');
+var cleanNpmMetadata = require('normalize-registry-metadata');
 var urllib = require('../common/urllib');
 var config = require('../config');

@@ -34,7 +20,11 @@ function* request(url, options) {
  url = registry + url;
  var r;
  try {
-    r = yield urllib.requestThunk(url, options);
+    r = yield urllib.request(url, options);
+    // https://github.com/npm/registry/issues/87#issuecomment-261450090
+    if (options.dataType === 'json' && r.data && config.officialNpmReplicate === registry) {
+      cleanNpmMetadata(r.data);
+    }
  } catch (err) {
    var statusCode = err.status || -1;
    var data = err.data || '[empty]';
--- a/services/package.js
+++ b/services/package.js
@@ -1,17 +1,5 @@
-/**
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var semver = require('semver');
 var models = require('../models');
 var common = require('./common');
@@ -75,7 +63,7 @@ exports.getModuleByTag = function* (name, tag) {
 };

 exports.getModuleByRange = function* (name, range) {
-  var rows = yield exports.listModulesByName(name);
+  var rows = yield exports.listModulesByName(name, [ 'id', 'version' ]);
  var versionMap = {};
  var versions = rows.map(function(row) {
    versionMap[row.version] = row;
@@ -85,7 +73,12 @@ exports.getModuleByRange = function* (name, range) {
  });

  var version = semver.maxSatisfying(versions, range);
-  return versionMap[version];
+  if (!versionMap[version]) {
+    return null;
+  }
+
+  var id = versionMap[version].id;
+  return yield exports.getModuleById(id);
 };

 exports.getLatestModule = function* (name) {
@@ -212,7 +205,7 @@ exports.listPublicModuleNamesByUser = function* (username) {
 };

 // start must be a date or timestamp
-exports.listPublicModuleNamesSince = function* (start) {
+exports.listPublicModuleNamesSince = function* listPublicModuleNamesSince(start) {
  if (!(start instanceof Date)) {
    start = new Date(Number(start));
  }
@@ -241,12 +234,13 @@ exports.listAllPublicModuleNames = function* () {
  });
 };

-exports.listModulesByName = function* (moduleName) {
+exports.listModulesByName = function* (moduleName, attributes) {
  var mods = yield Module.findAll({
    where: {
      name: moduleName
    },
-    order: [ ['id', 'DESC'] ]
+    order: [ ['id', 'DESC'] ],
+    attributes,
  });

  for (var mod of mods) {
@@ -339,7 +333,7 @@ exports.updateModulePackage = function* (id, pkg) {
 };

 exports.updateModulePackageFields = function* (id, fields) {
-  var mod = yield* exports.getModuleById(id);
+  var mod = yield exports.getModuleById(id);
  if (!mod) {
    return null;
  }
@@ -347,7 +341,7 @@ exports.updateModulePackageFields = function* (id, fields) {
  for (var k in fields) {
    pkg[k] = fields[k];
  }
-  return yield* exports.updateModulePackage(id, pkg);
+  return yield exports.updateModulePackage(id, pkg);
 };

 exports.updateModuleReadme = function* (id, readme) {
--- a/sync/changes_stream_syncer.js
+++ b/sync/changes_stream_syncer.js
@@ -0,0 +1,85 @@
+'use strict';
+
+const ChangesStream = require('changes-stream');
+const path = require('path');
+const fs = require('mz/fs');
+const urllib = require('urllib');
+const streamAwait = require('await-event');
+const logger = require('../common/logger');
+const config = require('../config');
+
+const db = config.officialNpmReplicate;
+const lastSeqFile = path.join(config.dataDir, '.cnpmjs.org.last_seq.txt');
+let _STREAM_ID = 0;
+
+module.exports = function* sync() {
+  const since = yield getLastSequence();
+  const streamId = _STREAM_ID++;
+  let changesCount = 0;
+  logger.syncInfo('start changes stream#%d, since: %s', streamId, since);
+  const changes = new ChangesStream({
+    db,
+    since,
+    include_docs: false,
+  });
+  changes.await = streamAwait;
+  changes.on('data', change => {
+    changesCount++;
+    logger.syncInfo('stream#%d get change#%d: %j', streamId, changesCount, change);
+    syncPackage(change);
+  });
+
+  try {
+    yield changes.await('error');
+  } catch (err) {
+    // make sure changes steam is destroy
+    changes.destroy();
+    err.message += `, stream#${streamId}, changesCount#${changesCount}`;
+    throw err;
+  }
+};
+
+function syncPackage(change) {
+  const url = `${config.handleSyncRegistry}/${change.id}/sync`;
+  urllib.request(url, {
+    method: 'PUT',
+    dataType: 'json',
+    timeout: 10000,
+  }, (err, data) => {
+    if (err) {
+      logger.syncInfo('%s:%s PUT %s error: %s, retry after 5s',
+        change.seq, change.id, url, err);
+      logger.syncError(err);
+      syncPackage(change);
+      setTimeout(() => syncPackage(change), 5000);
+    } else {
+      saveLastSequence(change.seq);
+      logger.syncInfo('%s:%s sync request sent, log: %s/log/%s',
+        change.seq, change.id, url, data.logId);
+    }
+  });
+}
+
+function* getLastSequence() {
+  let lastSeq;
+  if (yield fs.exists(lastSeqFile)) {
+    lastSeq = yield fs.readFile(lastSeqFile, 'utf8');
+    lastSeq = Number(lastSeq);
+  }
+  if (!lastSeq) {
+    lastSeq = 2649694;
+  }
+  // const r = yield urllib.request(db, {
+  //   dataType: 'json',
+  //   timeout: 15000,
+  // });
+  // logger.syncInfo('get registry info: %j', r.data);
+  // if (lastSeq < r.data.update_seq) {
+  //   lastSeq = r.data.update_seq;
+  // }
+  return lastSeq;
+}
+
+function saveLastSequence(seq) {
+  fs.writeFile(lastSeqFile, String(seq), () => {});
+}
--- a/sync/index.js
+++ b/sync/index.js
@@ -1,36 +1,23 @@
-/**
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
+const debug = require('debug')('cnpmjs.org:sync:index');
+const co = require('co');
+const ms = require('humanize-ms');
+const util = require('util');
+const config = require('../config');
+const mail = require('../common/mail');
+const logger = require('../common/logger');
+const totalService = require('../services/total');

-var debug = require('debug')('cnpmjs.org:sync:index');
-var co = require('co');
-var ms = require('humanize-ms');
-var util = require('util');
-var config = require('../config');
-var mail = require('../common/mail');
-var logger = require('../common/logger');
-var totalService = require('../services/total');
-
-var sync = null;
+let sync = null;

 switch (config.syncModel) {
-case 'all':
-  sync = require('./sync_all');
-  break;
-case 'exist':
-  sync = require('./sync_exist');
-  break;
+  case 'all':
+    sync = require('./sync_all');
+    break;
+  case 'exist':
+    sync = require('./sync_exist');
+    break;
 }

 if (!sync && config.enableCluster) {
@@ -38,34 +25,72 @@ if (!sync && config.enableCluster) {
  process.exit(0);
 }

-console.log('[%s] [sync_worker:%s] syncing with %s mode',
-  Date(), process.pid, config.syncModel);
+console.log('[%s] [sync_worker:%s] syncing with %s mode, changesStreamingSync: %s',
+  Date(), process.pid, config.syncModel, config.changesStreamingSync);

 function onerror(err) {
  logger.error(err);
 }

-//set sync_status = 0 at first
+// set sync_status = 0 at first
 co(function* () {
  yield totalService.updateSyncStatus(0);
  yield checkSyncStatus();
 }).catch(onerror);

-var syncInterval = ms(config.syncInterval);
-var minSyncInterval = ms('5m');
+let syncInterval = ms(config.syncInterval);
+const minSyncInterval = ms('5m');
 if (!syncInterval || syncInterval < minSyncInterval) {
  syncInterval = minSyncInterval;
 }

-// the same time only sync once
-var syncing = false;
-var syncFn = co.wrap(function* () {
-  debug('mode: %s, syncing: %s', config.syncModel, syncing);
-  if (!syncing) {
+if (sync) {
+  // the same time only sync once
+  let syncing = false;
+  const syncFn = co.wrap(function*() {
+    debug('mode: %s, syncing: %s', config.syncModel, syncing);
+    if (!syncing) {
+      syncing = true;
+      debug('start syncing');
+      let data;
+      let error;
+      try {
+        data = yield sync();
+      } catch (err) {
+        error = err;
+        error.message += ' (sync package error)';
+        logger.syncError(error);
+      }
+      data && logger.syncInfo(data);
+      if (!config.debug) {
+        sendMailToAdmin(error, data, new Date());
+      }
+      syncing = false;
+    }
+
+    // check last_sync_time and last_exist_sync_time
+    yield checkSyncStatus();
+  });
+
+  syncFn().catch(onerror);
+  setInterval(() => syncFn().catch(onerror), syncInterval);
+}
+
+/**
+ * sync popular modules
+ */
+
+if (config.syncPopular) {
+  const sync = require('./sync_popular');
+  let syncing = false;
+  const syncFn = co.wrap(function*() {
+    if (syncing) {
+      return;
+    }
    syncing = true;
-    debug('start syncing');
-    var data;
-    var error;
+    logger.syncInfo('Start syncing popular modules...');
+    let data;
+    let error;
    try {
      data = yield sync();
    } catch (err) {
@@ -73,79 +98,60 @@ var syncFn = co.wrap(function* () {
      error.message += ' (sync package error)';
      logger.syncError(error);
    }
-    data && logger.syncInfo(data);
+
+    if (data) {
+      logger.syncInfo(data);
+    }
    if (!config.debug) {
      sendMailToAdmin(error, data, new Date());
    }
    syncing = false;
-  }
+  });

-  // check last_sync_time and last_exist_sync_time
-  yield checkSyncStatus();
-});
-
-if (sync) {
  syncFn().catch(onerror);
-  setInterval(function () {
-    syncFn().catch(onerror);
-  }, syncInterval);
+  setInterval(() => syncFn().catch(onerror), ms(config.syncPopularInterval));
 }

-/**
- * sync popular modules
- */
+if (config.syncChangesStream) {
+  const sync = require('./changes_stream_syncer');
+  let syncing = false;
+  const syncFn = co.wrap(function*() {
+    if (syncing) {
+      return;
+    }
+    syncing = true;
+    logger.syncInfo('Start changes stream syncing...');
+    try {
+      yield sync();
+    } catch (err) {
+      err.message += ' (sync changes stream error)';
+      logger.syncError(err);
+    }
+    syncing = false;
+  });

-var startSyncPopular = require('./sync_popular');
-var syncingPopular = false;
-var syncPopularFn = co.wrap(function* syncPopular() {
-  if (syncingPopular) {
-    return;
-  }
-  syncingPopular = true;
-  logger.syncInfo('Start syncing popular modules...');
-  var data;
-  var error;
-  try {
-    data = yield startSyncPopular();
-  } catch (err) {
-    error = err;
-    error.message += ' (sync package error)';
-    logger.syncError(error);
-  }
-
-  if (data) {
-    logger.syncInfo(data);
-  }
-  if (!config.debug) {
-    sendMailToAdmin(error, data, new Date());
-  }
-
-  syncingPopular = false;
-});
-
-if (config.syncPopular) {
-  syncPopularFn().catch(onerror);
-  setInterval(function () {
-    syncPopularFn().catch(onerror);
-  }, ms(config.syncPopularInterval));
-} else {
-  logger.syncInfo('sync popular module disable');
+  syncFn().catch(onerror);
+  setInterval(() => syncFn().catch(onerror), ms('1m'));
 }

 function sendMailToAdmin(err, result, syncTime) {
  result = result || {};
-  var to = [];
+  const to = [];
  for (var name in config.admins) {
    to.push(config.admins[name]);
  }
  debug('Send email to all admins: %j, with err message: %s, result: %j, start sync time: %s.',
    to, err ? err.message : '', result, syncTime);
-  var subject;
-  var type;
-  var html;
+  let subject;
+  let type;
+  let html;
  if (err) {
-    // ignore 503 error
-    if (err.status === 503) {
+    // ignore 503, 504 error
+    // 504: Gateway Time-out
+    if (err.status === 503 || err.status === 504) {
+      return;
+    }
+    if (err.name === 'JSONResponseFormatError') {
      return;
    }
    subject = 'Sync Error';
@@ -159,6 +165,10 @@ function sendMailToAdmin(err, result, syncTime) {
      'Start sync time is %s.\n %d packges sync failed: %j ...\n %d packages sync successes :%j ...',
      syncTime, result.fails.length, result.fails.slice(0, 10),
      result.successes.length, result.successes.slice(0, 10));
+    // skip email notice when fails items small then 3
+    if (result.fails.length < 3) {
+      type = 'log';
+    }
  } else if (result.successes && result.successes.length) {
    subject = 'Sync Finished';
    type = 'log';
@@ -169,7 +179,7 @@ function sendMailToAdmin(err, result, syncTime) {
  debug('send email with type: %s, subject: %s, html: %s', type, subject, html);
  logger.syncInfo('send email with type: %s, subject: %s, html: %s', type, subject, html);
  if (type && type !== 'log') {
-    mail[type](to, subject, html, function (err) {
+    mail[type](to, subject, html, err => {
      if (err) {
        logger.error(err);
      }
@@ -182,8 +192,8 @@ function* checkSyncStatus() {
  if (!config.sourceNpmRegistryIsCNpm) {
    return;
  }
-  var total = yield totalService.getTotalInfo();
-  var lastSyncTime;
+  const total = yield totalService.getTotalInfo();
+  let lastSyncTime;
  if (config.syncModel === 'all') {
    lastSyncTime = total.last_sync_time;
  } else if (config.syncModel === 'exist') {
@@ -193,11 +203,13 @@ function* checkSyncStatus() {
  if (!lastSyncTime) {
    return;
  }
-  var diff = Date.now() - lastSyncTime;
-  var oneDay = 3600000 * 24;
-  if (diff > oneDay) {
-    var err = new Error('Last sync time is expired in ' + diff + ' ms, lastSyncTime: ' + new Date(lastSyncTime));
-    err.name = 'SyncExpriedError';
+  const diff = Date.now() - lastSyncTime;
+  const oneDay = 3600000 * 24;
+  const maxTime = Math.max(oneDay, syncInterval * 2);
+  if (diff > maxTime) {
+    const err = new Error('Last sync time is expired in ' + diff + ' ms, lastSyncTime: ' +
+      new Date(lastSyncTime) + ', maxTime: ' + maxTime + ' ms');
+    err.name = 'SyncExpiredError';
    sendMailToAdmin(err, null, new Date());
  }
 }
--- a/sync/status.js
+++ b/sync/status.js
@@ -1,19 +1,5 @@
-/*!
- * cnpmjs.org - sync/status.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:sync:status');
 var co = require('co');
 var Total = require('../services/total');
@@ -36,7 +22,7 @@ Status.prototype.log = function (syncDone) {
    lastSyncModule: this.lastSyncModule,
  };
  co(function* () {
-    yield* Total.updateSyncNum(params);
+    yield Total.updateSyncNum(params);
  }).catch(function () {});
 };

--- a/sync/sync_all.js
+++ b/sync/sync_all.js
@@ -1,20 +1,5 @@
-/**!
- * cnpmjs.org - sync/sync_all.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- *  fengmk2 <m@fengmk2.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var thunkify = require('thunkify-wrap');
 var config = require('../config');
 var Status = require('./status');
@@ -72,19 +57,22 @@ module.exports = function* sync() {
    concurrency: config.syncConcurrency,
    syncUpstreamFirst: false,
  });
-  Status.init({need: packages.length}, worker);
+  Status.init({
+    need: packages.length,
+  }, worker);
  worker.start();
  var end = thunkify.event(worker);
  yield end();

-  logger.syncInfo('All packages sync done, successes %d, fails %d',
-      worker.successes.length, worker.fails.length);
+  logger.syncInfo('All packages sync done, successes %d, fails %d, updates %d',
+      worker.successes.length, worker.fails.length, worker.updates.length);
  //only when all succss, set last sync time
  if (!worker.fails.length) {
-    yield* totalService.setLastSyncTime(syncTime);
+    yield totalService.setLastSyncTime(syncTime);
  }
  return {
    successes: worker.successes,
-    fails: worker.fails
+    fails: worker.fails,
+    updates: worker.updates,
  };
 };
--- a/sync/sync_exist.js
+++ b/sync/sync_exist.js
@@ -1,17 +1,5 @@
-/*!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:sync:sync_exist');
 var Status = require('./status');
 var thunkify = require('thunkify-wrap');
@@ -31,7 +19,7 @@ function intersection(arrOne, arrTwo) {
    map[name] = true;
  });
  arrTwo.forEach(function (name) {
-    map[name] && results.push(name);
+    map[name] === true && results.push(name);
  });
  return results;
 }
@@ -48,7 +36,7 @@ module.exports = function* sync() {

  var allPackages;
  if (!info.last_exist_sync_time) {
-    var pkgs = yield* npmService.getShort();
+    var pkgs = yield npmService.getShort();
    debug('First time sync all packages from official registry, got %d packages', pkgs.length);
    if (info.last_sync_module) {
      // start from last success
@@ -74,7 +62,7 @@ module.exports = function* sync() {
      fails: []
    };
  }
-  debug('Total %d packages to sync', packages.length);
+  debug('Total %d packages to sync, top 10: %j', packages.length, packages.slice(0, 10));

  var worker = new SyncModuleWorker({
    username: 'admin',
--- a/sync/sync_popular.js
+++ b/sync/sync_popular.js
@@ -1,19 +1,5 @@
-/*!
- * cnpmjs.org - sync/sync_popular.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var debug = require('debug')('cnpmjs.org:sync:sync_popular');
 var thunkify = require('thunkify-wrap');
 var config = require('../config');
--- a/sync/sync_since.js
+++ b/sync/sync_since.js
@@ -1,19 +1,5 @@
-/**!
- * sync packages since by some days ago
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <m@fengmk2.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 const thunkify = require('thunkify-wrap');
 const co = require('co');
 const ms = require('humanize-ms');
--- a/test/controllers/registry/module/scope_package.test.js
+++ b/test/controllers/registry/module/scope_package.test.js
@@ -1,13 +1,3 @@
-/**!
- * cnpmjs.org - test/controllers/registry/module/scope_package.test.js
- *
- * Copyright(c) fengmk2 and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

 /**
@@ -21,7 +11,7 @@ var config = require('../../../../config');
 var app = require('../../../../servers/registry');
 var utils = require('../../../utils');

-describe('controllers/registry/module/scope_package.test.js', function () {
+describe('test/controllers/registry/module/scope_package.test.js', function () {
  var pkgname = '@cnpm/test-scope-package';
  var pkgURL = '/@' + encodeURIComponent(pkgname.substring(1));
  before(function (done) {
@@ -52,16 +42,17 @@ describe('controllers/registry/module/scope_package.test.js', function () {

  afterEach(mm.restore);

-  it('should get 404 when do not support scope', function (done) {
+  it('should get 302 when do not support scope', function (done) {
    mm(config, 'scopes', []);
    request(app)
    .get('/@invalid/test')
-    .expect(404, done);
+    .expect('Location', 'https://registry.npmjs.com/@invalid/test')
+    .expect(302, done);
  });

-  it('should get 400 when scope not match', function (done) {
+  it('should get 404 when scope is private', function (done) {
    request(app)
-    .get('/@invalid/test')
+    .get('/@cnpmtest/test')
    .expect(404, done);
  });

--- a/test/controllers/registry/package/download_total.test.js
+++ b/test/controllers/registry/package/download_total.test.js
@@ -1,38 +1,34 @@
-/**!
- * cnpmjs.org - test/controllers/registry/package/download_total.test.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   dead_horse <dead_horse@qq.com> (https://github.com/dead-horse)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
+const request = require('supertest');
+const mm = require('mm');
+const DownloadTotal = require('../../../../services/download_total');
+const app = require('../../../../servers/registry');
+const utils = require('../../../utils');

-var request = require('supertest');
-var mm = require('mm');
-var DownloadTotal = require('../../../../services/download_total');
-var app = require('../../../../servers/registry');
-
-describe('controllers/registry/package/download_total.test.js', function () {
+describe('test/controllers/registry/package/download_total.test.js', () => {
  afterEach(mm.restore);

-  it('should error when range error', function (done) {
-    request(app.listen())
+  before(() => {
+    const pkg2 = utils.getPackage('@cnpmtest/download_total_test_module', '1.0.1', utils.otherUser);
+    return request(app.listen())
+      .put('/' + pkg2.name)
+      .set('authorization', utils.otherUserAuth)
+      .send(pkg2)
+      .expect(201);
+  });
+
+  it('should error when range error', () => {
+    return request(app.listen())
    .get('/downloads/range/2014-10-10:xxxx/koa')
    .expect(400)
    .expect({
      error: 'range_error',
      reason: 'range must be YYYY-MM-DD:YYYY-MM-DD style'
-    }, done);
+    });
  });

-  it('should get package downloads ok', function (done) {
+  it('should get package downloads ok', () => {
    mm.data(DownloadTotal, 'getModuleTotal', [{
      id: 1,
      count: 10,
@@ -50,7 +46,7 @@ describe('controllers/registry/package/download_total.test.js', function () {
      name: 'koa'
    }]);

-    request(app.listen())
+    return request(app.listen())
    .get('/downloads/range/2014-12-01:2014-12-03/koa')
    .expect(200)
    .expect({
@@ -67,10 +63,10 @@ describe('controllers/registry/package/download_total.test.js', function () {
        day: '2014-12-03',
        downloads: 10
      }]
-    }, done);
+    });
  });

-  it('should get total downloads ok', function (done) {
+  it('should get total downloads ok', () => {
    mm.data(DownloadTotal, 'getTotal', [{
      count: 20,
      date: '2014-12-03',
@@ -82,7 +78,7 @@ describe('controllers/registry/package/download_total.test.js', function () {
      date: '2014-12-02',
    }]);

-    request(app.listen())
+    return request(app.listen())
    .get('/downloads/range/2014-12-01:2014-12-03')
    .expect(200)
    .expect({
@@ -98,6 +94,44 @@ describe('controllers/registry/package/download_total.test.js', function () {
        day: '2014-12-03',
        downloads: 20
      }]
-    }, done);
+    });
+  });
+
+  it('should get scope package downloads ok', () => {
+    mm.data(DownloadTotal, 'getModuleTotal', [{
+      id: 1,
+      count: 10,
+      date: '2014-12-03',
+      name: '@cnpmtest/download_total_test_module'
+    }, {
+      id: 1,
+      count: 8,
+      date: '2014-12-01',
+      name: '@cnpmtest/download_total_test_module'
+    }, {
+      id: 1,
+      count: 5,
+      date: '2014-12-02',
+      name: '@cnpmtest/download_total_test_module'
+    }]);
+
+    return request(app.listen())
+    .get('/downloads/range/2014-12-01:2014-12-03/@cnpmtest/download_total_test_module')
+    .expect(200)
+    .expect({
+      start: '2014-12-01',
+      end: '2014-12-03',
+      package: '@cnpmtest/download_total_test_module',
+      downloads: [{
+        day: '2014-12-01',
+        downloads: 8
+      }, {
+        day: '2014-12-02',
+        downloads: 5
+      }, {
+        day: '2014-12-03',
+        downloads: 10
+      }]
+    });
  });
 });
--- a/test/controllers/registry/package/list.test.js
+++ b/test/controllers/registry/package/list.test.js
@@ -1,19 +1,5 @@
-/**!
- * cnpmjs.org - test/controllers/registry/package/list.test.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var should = require('should');
 var request = require('supertest');
 var mm = require('mm');
@@ -23,7 +9,7 @@ var app = require('../../../../servers/registry');
 var utils = require('../../../utils');
 var config = require('../../../../config');

-describe('controllers/registry/package/list.test.js', function () {
+describe('test/controllers/registry/package/list.test.js', () => {
  afterEach(mm.restore);

  before(function (done) {
@@ -85,6 +71,7 @@ describe('controllers/registry/package/list.test.js', function () {
        fengmk2: true,
        foouser: true
      });
+      data.versions['0.0.1'].publish_time.should.equal(data.versions['0.0.1']._cnpm_publish_time);
      done();
    });
  });
@@ -135,12 +122,13 @@ describe('controllers/registry/package/list.test.js', function () {
    }, done);
  });

-  describe('unpublished', function () {
-    before(function (done) {
+  describe.skip('unpublished', () => {
+    before(done => {
+      mm(config, 'syncModel', 'all');
      utils.sync('moduletest1', done);
    });

-    it('should show unpublished info', function (done) {
+    it('should show unpublished info', done => {
      mm(config, 'syncModel', 'all');
      request(app.listen())
      .get('/moduletest1')
--- a/test/controllers/registry/package/show.test.js
+++ b/test/controllers/registry/package/show.test.js
@@ -1,17 +1,5 @@
-/**
- * Copyright(c) cnpm and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var should = require('should');
 var request = require('supertest');
 var mm = require('mm');
@@ -55,7 +43,11 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('0.0.1');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      data.dist.tarball.should.containEql('/@cnpmtest/testmodule-show/download/@cnpmtest/testmodule-show-0.0.1.tgz');
+      data._cnpm_publish_time.should.equal(data.publish_time);
      done();
    });
  });
@@ -68,6 +60,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('1.1.0');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      data.dist.tarball.should.containEql('/@cnpmtest/testmodule-show/download/@cnpmtest/testmodule-show-1.1.0.tgz');
      done();
    });
@@ -81,6 +76,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('0.0.1');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      data.dist.tarball.should.containEql('/@cnpmtest/testmodule-show/download/@cnpmtest/testmodule-show-0.0.1.tgz');
      done();
    });
@@ -94,6 +92,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('1.1.0');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      data.dist.tarball.should.containEql('/@cnpmtest/testmodule-show/download/@cnpmtest/testmodule-show-1.1.0.tgz');
      done();
    });
@@ -107,6 +108,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-only-beta');
      data.version.should.equal('1.0.0-beta.1');
+      data['dist-tags'].should.eql({
+        latest: '1.0.0-beta.1',
+      });
      data.dist.tarball.should.containEql('/@cnpmtest/testmodule-only-beta/download/@cnpmtest/testmodule-only-beta-1.0.0-beta.1.tgz');
      done();
    });
@@ -127,6 +131,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('1.1.0');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      done();
    });
  });
@@ -145,6 +152,9 @@ describe('test/controllers/registry/package/show.test.js', function () {
      var data = res.body;
      data.name.should.equal('@cnpmtest/testmodule-show');
      data.version.should.equal('0.0.1');
+      data['dist-tags'].should.eql({
+        latest: '1.1.0',
+      });
      done();
    });
  });
--- a/test/controllers/sync_module_worker.test.js
+++ b/test/controllers/sync_module_worker.test.js
@@ -1,17 +1,5 @@
-/**!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var should = require('should');
 var mm = require('mm');
 var thunkify = require('thunkify-wrap');
@@ -65,10 +53,12 @@ describe('test/controllers/sync_module_worker.test.js', function () {
  });

  it('should sync public scoped package', function* () {
+    mm(config, 'registryHost', '');
    mm(config, 'sourceNpmRegistry', 'https://registry.npmjs.org');
    var worker = new SyncModuleWorker({
      name: '@sindresorhus/df',
      username: 'fengmk2',
+      noDep: true,
    });
    worker.start();
    var end = thunkify.event(worker, 'end');
@@ -99,6 +89,7 @@ describe('test/controllers/sync_module_worker.test.js', function () {
    yield checkResult();

    var r = yield urllib.request(tgzUrl);
+    console.log(r.status, r.headers);
    r.status.should.equal(200);
  });

@@ -190,7 +181,7 @@ describe('test/controllers/sync_module_worker.test.js', function () {

  it('should sync unpublished info', function (done) {
    var worker = new SyncModuleWorker({
-      name: ['tnpm'],
+      name: ['afp'],
      username: 'fengmk2'
    });

@@ -198,7 +189,7 @@ describe('test/controllers/sync_module_worker.test.js', function () {
    worker.on('end', function () {
      var names = worker.successes.concat(worker.fails);
      names.sort();
-      names.should.eql(['tnpm']);
+      names.should.eql([ 'afp' ]);
      done();
    });
  });
@@ -288,6 +279,44 @@ describe('test/controllers/sync_module_worker.test.js', function () {
    });
  });

+  describe('sync deprecated info', () => {
+    before(function* () {
+      mm(config, 'syncModel', 'all');
+      const worker = new SyncModuleWorker({
+        name: 'ms',
+        username: 'fengmk2',
+      });
+      worker.start();
+      const end = thunkify.event(worker, 'end');
+      yield end();
+    });
+
+    it('should sync support un-deprecate action', function* () {
+      const listModulesByName = packageService.listModulesByName;
+      mm(packageService, 'listModulesByName', function* (name) {
+        const mods = yield listModulesByName.call(packageService, name);
+        mods.forEach(function (mod) {
+          mod.package.deprecated = 'mock deprecated';
+        });
+        return mods;
+      });
+
+      var worker = new SyncModuleWorker({
+        name: 'ms',
+        username: 'fengmk2',
+      });
+      worker.start();
+      const end = thunkify.event(worker, 'end');
+      yield end();
+      mm.restore();
+      // check deprecated
+      const mods = yield packageService.listModulesByName('ms');
+      for (const mod of mods) {
+        should.ok(mod.package.deprecated === undefined);
+      }
+    });
+  });
+
  describe('sync user', function () {
    it('should sync fengmk2', function* () {
      var worker = new SyncModuleWorker({
--- a/test/controllers/web/package/search.test.js
+++ b/test/controllers/web/package/search.test.js
@@ -22,6 +22,9 @@ var registry = require('../../../../servers/registry');
 var utils = require('../../../utils');

 describe('controllers/web/package/search.test.js', function () {
+
+  var app_reg;
+
  before(function (done) {
    var pkg = utils.getPackage('@cnpmtest/testmodule-web-search', '0.0.1', utils.admin);
    pkg.versions['0.0.1'].dependencies = {
@@ -29,7 +32,10 @@ describe('controllers/web/package/search.test.js', function () {
      mocha: '~1.0.0',
      'testmodule-web-show': '0.0.1'
    };
-    request(registry.listen())
+
+    app_reg = registry.listen();
+    
+    request(app_reg)
    .put('/' + pkg.name)
    .set('authorization', utils.adminAuth)
    .send(pkg)
@@ -73,5 +79,51 @@ describe('controllers/web/package/search.test.js', function () {
      .expect(200)
      .expect(/Can not found package match notexistpackage/, done);
    });
+
+
+    describe('GET /browse/keyword/:word searchlist', function () {
+
+      before(function (done) {
+        var pkg = utils.getPackage('@cnpmtest/testmodule-web-search_a', '0.0.1', utils.admin);
+        pkg.versions['0.0.1'].dependencies = {
+          bytetest: '~0.0.1',
+          mocha: '~1.0.0',
+          'testmodule-web-show': '0.0.1'
+        };
+        request(app_reg)
+          .put('/' + pkg.name)
+          .set('authorization', utils.adminAuth)
+          .send(pkg)
+          .expect(201, done);
+      });
+
+      it('should list by keyword with json(default limit=100)', function (done) {
+          request(app)
+          .get('/browse/keyword/@cnpmtest/testmodule-web-searc?type=json')
+          .expect(200)
+          .expect({
+            keyword: '@cnpmtest/testmodule-web-searc',
+            match: null,
+            packages: [ { name: '@cnpmtest/testmodule-web-search', description: '' },
+            { name: '@cnpmtest/testmodule-web-search_a', description: '' }],
+            keywords: []
+          })
+          .expect('content-type', 'application/json; charset=utf-8', done);
+      });
+
+      it('should list by keyword with json(use limit)', function (done) {
+          request(app)
+          .get('/browse/keyword/@cnpmtest/testmodule-web-searc?type=json&limit=1')
+          .expect(200)
+          .expect({
+            keyword: '@cnpmtest/testmodule-web-searc',
+            match: null,
+            packages: [ { name: '@cnpmtest/testmodule-web-search', description: '' }],
+            keywords: []
+          })
+          .expect('content-type', 'application/json; charset=utf-8', done);
+      });
+
+    });
  });
 });
--- a/test/controllers/web/package/show.test.js
+++ b/test/controllers/web/package/show.test.js
@@ -1,19 +1,5 @@
-/*!
- * cnpmjs.org - test/controllers/web/package/show.test.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com> (http://deadhorse.me)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var should = require('should');
 var request = require('supertest');
 var mm = require('mm');
@@ -22,7 +8,7 @@ var app = require('../../../../servers/web');
 var registry = require('../../../../servers/registry');
 var utils = require('../../../utils');

-describe('controllers/web/package/show.test.js', function () {
+describe('test/controllers/web/package/show.test.js', () => {
  before(function (done) {
    var pkg = utils.getPackage('@cnpmtest/testmodule-web-show', '0.0.1', utils.admin);
    pkg.versions['0.0.1'].dependencies = {
@@ -50,7 +36,7 @@ describe('controllers/web/package/show.test.js', function () {
      .expect(/Dependencies/)
      .expect(/Downloads/, function (err, res) {
        should.not.exist(err);
-        res.should.have.header('etag');
+        should.exist(res.headers.etag);
        res.text.should.containEql('<meta charset="utf-8">');
        done();
      });
@@ -66,7 +52,7 @@ describe('controllers/web/package/show.test.js', function () {
      .expect(/Dependencies/)
      .expect(/Downloads/, function (err, res) {
        should.not.exist(err);
-        res.should.have.header('etag');
+        should.exist(res.headers.etag);
        res.text.should.containEql('<meta charset="utf-8">');
        done();
      });
@@ -113,23 +99,25 @@ describe('controllers/web/package/show.test.js', function () {
    });
  });

-  describe('unpublished package', function () {
-    before(function (done) {
-      utils.sync('tfs', done);
+  describe.skip('unpublished package', () => {
+    before(done => {
+      mm(config, 'syncModel', 'all');
+      utils.sync('mk2testmodule', done);
    });

-    it('should display unpublished info', function (done) {
+    it('should display unpublished info', () => {
      mm(config, 'syncModel', 'all');
-      request(app)
-      .get('/package/moduletest1')
-      .expect(200)
-      .expect(/This package has been unpublished\./, done);
+      return request(app)
+        .get('/package/mk2testmodule')
+        .expect(200)
+        .expect(/This package has been unpublished\./);
    });
  });

  describe('xss filter', function () {
    before(function (done) {
-      var pkg = utils.getPackage('@cnpmtest/xss-test-ut', '0.0.1', utils.admin, null, '[xss link](javascript:alert(2)) \n\nfoo<script>alert(1)</script>/xss\'"&#');
+      var pkg = utils.getPackage('@cnpmtest/xss-test-ut', '0.0.1', utils.admin,
+        null, '[xss link](javascript:alert(2)) \n\nfoo<script>alert(1)</script>/xss\'"&#');
      request(registry.listen())
      .put('/' + pkg.name)
      .set('authorization', utils.adminAuth)
@@ -149,20 +137,214 @@ describe('controllers/web/package/show.test.js', function () {
    });
  });

-  describe('show npm package', function () {
-    before(function (done) {
+  describe('show npm package', () => {
+    before(done => {
      mm(config, 'syncModel', 'exists');
      utils.sync('pedding', done);
    });

-    it('should show pedding package info and contributors', function (done) {
+    it('should show pedding package info and contributors', () => {
      mm(config, 'syncModel', 'exists');
-      request(app)
-      .get('/package/pedding')
+      return request(app)
+        .get('/package/pedding')
+        .expect(200)
+        // https://github.com/cnpm/cnpmjs.org/issues/497
+        .expect(/by <a href="\/\~fengmk2">fengmk2<\/a>/)
+        // snyk link
+        .expect(/<a class="badge-link" href="https:\/\/snyk\.io\/test\/npm\/pedding" target="_blank"><img title="Known Vulnerabilities" src="https:\/\/snyk\.io\/test\/npm\/pedding\/badge\.svg\?style=flat-square"><\/a>/)
+        .expect(/pedding/);
+    });
+  });
+
+  describe('show repository url in git syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-git', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'git://github.com/cnpm/cnpmjs.org.git'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-git')
      .expect(200)
-      // https://github.com/cnpm/cnpmjs.org/issues/497
-      .expect(/by <a href="\/\~fengmk2">fengmk2<\/a>/)
-      .expect(/pedding/, done);
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-git/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/https:\/\/github\.com\/cnpm\/cnpmjs\.org/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
+    });
+  });
+
+   describe('show repository url in ssh syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-ssh', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'git@github.com:cnpm/cnpmjs.org.git'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-ssh')
+      .expect(200)
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-ssh/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/https:\/\/github\.com\/cnpm\/cnpmjs\.org/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
+    });
+  });
+
+  describe('show repository url in raw ssh syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-raw-ssh', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'ssh://git@github.com/cnpm/cnpmjs.org.git'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-raw-ssh')
+      .expect(200)
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-raw-ssh/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/https:\/\/github\.com\/cnpm\/cnpmjs\.org/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
+    });
+  });
+
+  describe('show repository url in https syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-https', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'https://github.com/cnpm/cnpmjs.org.git'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-https')
+      .expect(200)
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-https/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/https:\/\/github\.com\/cnpm\/cnpmjs\.org\.git/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
+    });
+  });
+
+   describe('show repository url in short https syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-short-https', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'https://github.com/cnpm/cnpmjs.org'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-short-https')
+      .expect(200)
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-short-https/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/https:\/\/github\.com\/cnpm\/cnpmjs\.org/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
+    });
+  });
+
+  describe('show repository url in short http syntax', () => {
+    before(function (done) {
+      var pkg = utils.getPackage('@cnpmtest/testmodule-repo-short-http', '0.0.1', utils.admin);
+      pkg.versions['0.0.1'].repository = {
+        type: 'git',
+        url: 'http://github.com/cnpm/cnpmjs.org.git'
+      }
+      request(registry.listen())
+      .put('/' + pkg.name)
+      .set('authorization', utils.adminAuth)
+      .send(pkg)
+      .expect(201, done);
+    });
+
+    it('should get 200', function (done) {
+      request(app.listen())
+      .get('/package/@cnpmtest/testmodule-repo-short-http')
+      .expect(200)
+      .expect('content-type', 'text/html; charset=utf-8')
+      .expect(/testmodule-repo-short-http/)
+      .expect(/Maintainers/)
+      .expect(/Dependencies/)
+      .expect(/http:\/\/github\.com\/cnpm\/cnpmjs\.org/)
+      .expect(/Downloads/, function (err, res) {
+        should.not.exist(err);
+        should.exist(res.headers.etag);
+        res.text.should.containEql('<meta charset="utf-8">');
+        done();
+      });
    });
  });
 });
--- a/test/init.js
+++ b/test/init.js
@@ -1,13 +1,3 @@
-/**!
- * cnpmjs.org - test/init.js
- *
- * Copyright(c) fengmk2 and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

 /**
--- a/test/init_db.js
+++ b/test/init_db.js
@@ -1,13 +1,3 @@
-/**!
- * cnpmjs.org - test/init_db.js
- *
- * Copyright(c) fengmk2 and other contributors.
- * MIT Licensed
- *
- * Authors:
- *   fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

 /**
@@ -22,7 +12,7 @@ var config = require('../config');

 // init db first
 var initscript = path.join(__dirname, '..', 'models', 'init_script.js');
-var cmd = ['node', '--harmony', initscript, 'true',
+var cmd = ['node', initscript, 'true',
  config.database.dialect, config.database.port, config.database.username].join(' ');
 console.log('$ %s', cmd);
 var stdout = childProcess.execSync(cmd);
@@ -35,6 +25,7 @@ var usernames = [
  'cnpmjstest101',
  'cnpmjstest102',
  'cnpmjstest103',
+  ['cnpmjstest104', 'cnpmjs:test104'],
  'cnpmjstest10', // admin
  'cnpmjstestAdmin2', // other admin
  'cnpmjstestAdmin3', // other admin
@@ -43,6 +34,13 @@ var usernames = [

 var count = usernames.length;
 usernames.forEach(function (name) {
+  var pass;
+  if (Array.isArray(name)) {
+    name = name[0];
+    pass = name[1];
+  } else {
+    pass = name;
+  }
  var user = User.build({
    name: name,
    email: 'fengmk2@gmail.com',
@@ -50,7 +48,7 @@ usernames.forEach(function (name) {
    rev: '1',
  });
  user.salt = crypto.randomBytes(30).toString('hex');
-  user.password_sha = User.createPasswordSha(name, user.salt);
+  user.password_sha = User.createPasswordSha(pass, user.salt);
  user.save().then(function () {
    count--;
    if (count === 0) {
--- a/test/middleware/auth.test.js
+++ b/test/middleware/auth.test.js
@@ -53,6 +53,13 @@ describe('middleware/auth.test.js', function () {
      .expect(200, done);
    });

+    it('should pass with authorization (password contains ":") and check ok', function (done) {
+      request(app)
+      .get('/-/user/org.couchdb.user:cnpmjstest104')
+      .set('authorization', 'basic ' + new Buffer('cnpmjstest104:cnpmjs:test104').toString('base64'))
+      .expect(200, done);
+    });
+
    describe('config.customUserService = true', function () {
      beforeEach(function () {
        mm(config, 'customUserService', true);
--- a/test/middleware/proxy_to_npm.test.js
+++ b/test/middleware/proxy_to_npm.test.js
@@ -1,13 +1,3 @@
-/**!
- * cnpmjs.org - test/middleware/proxy_to_npm.test.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

 /**
@@ -19,44 +9,51 @@ var mm = require('mm');
 var app = require('../../servers/registry');
 var config = require('../../config');

-describe('middleware/proxy_to_npm.test.js', function () {
-  beforeEach(function () {
+describe('test/middleware/proxy_to_npm.test.js', () => {
+  beforeEach(() => {
    mm(config, 'syncModel', 'none');
  });

  afterEach(mm.restore);

-  describe('package', function () {
-    it('should proxy to source registry when package not exists', function (done) {
+  describe('package', () => {
+    it('should proxy to source registry when package not exists', done => {
      request(app.listen())
      .get('/ms')
      .expect('location', config.sourceNpmRegistry + '/ms')
      .expect(302, done);
    });

-    it('should proxy to source registry when package is not local', function (done) {
+    it('should proxy to source registry when package is not local', done => {
      request(app.listen())
      .get('/baidu?write=true')
      .expect('location', config.sourceNpmRegistry + '/baidu?write=true')
      .expect(302, done);
    });

-    it('should not proxy to source registry when package is scoped', function (done) {
+    it('should not proxy to source registry when package is private scoped', done => {
      request(app.listen())
-      .get('/@scoped/test-package-name')
+      .get('/@cnpmtest/test-package-name')
      .expect(404, done);
    });
+
+    it('should proxy to source registry when package is public scoped', done => {
+      request(app.listen())
+      .get('/@jkroso/type')
+      .expect('Location', 'https://registry.npmjs.com/@jkroso/type')
+      .expect(302, done);
+    });
  });

-  describe('dist-tags', function () {
-    it('should proxy to source registry when package not exists', function (done) {
+  describe('dist-tags', () => {
+    it('should proxy to source registry when package not exists', done => {
      request(app.listen())
      .get('/-/package/ms/dist-tags')
      .expect('location', config.sourceNpmRegistry + '/-/package/ms/dist-tags')
      .expect(302, done);
    });

-    it('should dont proxy scoped package', function (done) {
+    it('should dont proxy scoped package', done => {
      request(app.listen())
      .get('/-/package/@scoped/ms/dist-tags')
      .expect(404, done);
--- a/test/middleware/publishable.test.js
+++ b/test/middleware/publishable.test.js
@@ -0,0 +1,116 @@
+'use strict';
+
+var app = require('../../servers/registry');
+var config = require('../../config');
+var request = require('supertest');
+var utils = require('../utils');
+var mm = require('mm');
+
+describe('middleware/publishable.test.js', function () {
+  afterEach(mm.restore);
+
+  it('should not 403 when admin put to public package', function (done) {
+    mm(config, 'enablePrivate', false);
+    request(app)
+    .put('/koa/-rev/1')
+    .set('authorization', utils.adminAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(404, done);
+  });
+
+  it('should not 403 when admin put to unsupported scope package', function (done) {
+    mm(config, 'enablePrivate', false);
+    request(app)
+    .put('/@unsupported/koa/-rev/1')
+    .set('authorization', utils.adminAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(404, done);
+  });
+
+  it('should not 403 when normal user put to white list package', function (done) {
+    mm(config, 'enablePrivate', false);
+    mm(config, 'privatePackages', ['koa'])
+    request(app)
+    .put('/koa/-rev/1')
+    .set('authorization', utils.otherUserAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(404, done);
+  });
+
+  it('should not 403 when normal user put to supported scope', function (done) {
+    mm(config, 'enablePrivate', false);
+    mm(config, 'scopes', ['@test'])
+    request(app)
+    .put('/@test/koa/-rev/1')
+    .set('authorization', utils.otherUserAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(404, done);
+  });
+
+  it('should 400 when normal user put to unsupported scope', function (done) {
+    mm(config, 'enablePrivate', false);
+    mm(config, 'scopes', ['@test'])
+    request(app)
+    .put('/@test1/koa/-rev/1')
+    .set('authorization', utils.otherUserAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(400, done);
+  });
+
+  it('should 403 when common user put to public package', function (done) {
+    mm(config, 'enablePrivate', false);
+    request(app)
+    .put('/koa/-rev/1')
+    .set('authorization', utils.otherUserAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect(403, done);
+  });
+
+  it('should 403 when common user put in private mode', function (done) {
+    mm(config, 'enablePrivate', true);
+    request(app)
+    .put('/koa/-rev/1')
+    .set('authorization', utils.otherUserAuth)
+    .send({
+      versions: {
+        '0.0.1': {},
+        '0.0.2': {}
+      }
+    })
+    .expect({
+      error: 'no_perms',
+      reason: 'Private mode enable, only admin can publish this module'
+    })
+    .expect(403, done);
+  });
+});
--- a/test/middleware/sync_by_install.test.js
+++ b/test/middleware/sync_by_install.test.js
@@ -0,0 +1,34 @@
+'use strict';
+
+/**
+ * Module dependencies.
+ */
+
+var request = require('supertest');
+var app = require('../../servers/registry');
+var mm = require('mm');
+var config = require('../../config');
+var userService = require('../../services/user');
+
+describe('test/middleware/sync_by_install.test.js', () => {
+  afterEach(mm.restore);
+
+  it('should ignore sync on install private scoped package', done => {
+    request(app.listen())
+    .get('/@cnpmtest/foo')
+    .set('User-Agent', 'node/v4.4.4')
+    .expect({
+      error: 'not_found',
+      reason: 'document not found',
+    })
+    .expect(404, done);
+  });
+
+  it('should sync and redirect to npmjs.com on install public scoped package', done => {
+    request(app.listen())
+    .get('/@jkroso/type')
+    .set('User-Agent', 'node/v4.4.4')
+    .expect('Location', 'https://registry.npmjs.com/@jkroso/type')
+    .expect(302, done);
+  });
+});
--- a/test/services/npm.test.js
+++ b/test/services/npm.test.js
@@ -1,45 +1,32 @@
-/**!
- * cnpmjs.org - test/services/npm.test.js
- *
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.github.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var should = require('should');
 var mm = require('mm');
 var fs = require('fs');
 var path = require('path');
 var ChunkStream = require('chunkstream');
+var config = require('../../config');
 var npm = require('../../services/npm');

 var fixtures = path.join(path.dirname(__dirname), 'fixtures');

-describe('services/npm.test.js', function () {
+describe('services/npm.test.js', () => {
  afterEach(mm.restore);

  it('should return a module info from source npm', function* () {
-    var data = yield* npm.get('pedding');
+    var data = yield npm.get('pedding');
    data.name.should.equal('pedding');
  });

  it('should return null when module not exist', function *() {
-    var data = yield* npm.get('pedding-not-exists');
+    var data = yield npm.get('pedding-not-exists');
    should.not.exist(data);
  });

  it.skip('should return error when http error', function* () {
    mm.http.request(/\//, new ChunkStream(['{']));
    try {
-      yield* npm.get('pedding-not-exists');
+      yield npm.get('pedding-not-exists');
      throw new Error('should not run this');
    } catch (err) {
      err.name.should.equal('JSONResponseFormatError');
@@ -51,7 +38,7 @@ describe('services/npm.test.js', function () {
    mm.http.request(/\//, content, { statusCode: 500 });
    // http://registry.npmjs.org/octopie
    try {
-      yield* npm.get('octopie');
+      yield npm.get('octopie');
      throw new Error('should not run this');
    } catch (err) {
      err.name.should.equal('NPMServerError');
@@ -60,7 +47,31 @@ describe('services/npm.test.js', function () {
    }
  });

-  describe('getPopular()', function () {
+  describe('request()', () => {
+    it('should request from replicate and clean meta data', function* () {
+      const result = yield npm.request('/shelljs', {
+        registry: config.officialNpmReplicate,
+      });
+      const pkg = result.data;
+      pkg.name.should.equal('shelljs');
+      pkg.time['0.0.1-alpha1'].should.equal('2012-03-02T21:46:14.725Z');
+      pkg.versions['0.0.1-alpha1'].version.should.equal('0.0.1-alpha1');
+      pkg.versions['0.0.1-alpha1'].dist.shasum.should.equal('cfa9394e29c3eb0fe58998f5bf5bda79aa7d3e2e');
+      pkg.versions['0.0.1-alpha1'].dist.tarball.should.equal('http://registry.npmjs.org/shelljs/-/shelljs-0.0.1alpha1.tgz');
+
+      pkg.time['0.7.5'].should.equal('2016-10-27T05:50:21.479Z');
+      pkg.versions['0.7.5'].version.should.equal('0.7.5');
+      pkg.versions['0.7.5'].dist.shasum.should.equal('2eef7a50a21e1ccf37da00df767ec69e30ad0675');
+      pkg.versions['0.7.5'].dist.tarball.should.equal('http://registry.npmjs.org/shelljs/-/shelljs-0.7.5.tgz');
+
+      pkg.time['0.0.6-pre2'].should.equal('2012-05-25T18:14:25.441Z');
+      pkg.versions['0.0.6-pre2'].version.should.equal('0.0.6-pre2');
+      pkg.versions['0.0.6-pre2'].dist.shasum.should.equal('8c3eecaddba6f425bd5cae001f80a4d224750911');
+      pkg.versions['0.0.6-pre2'].dist.tarball.should.equal('http://registry.npmjs.org/shelljs/-/shelljs-0.0.6pre2.tgz');
+    });
+  });
+
+  describe('getPopular()', () => {
    it('should return popular modules', function* () {
      mm.http.request(/\//, JSON.stringify({
        rows: [
@@ -82,7 +93,7 @@ describe('services/npm.test.js', function () {
          { key: ['foo15'], value: 1 },
        ]
      }));
-      var rows = yield* npm.getPopular(10);
+      var rows = yield npm.getPopular(10);
      rows.should.length(2);
      rows[0][0].should.equal('underscore');
    });
--- a/test/sync/sync_exist.test.js
+++ b/test/sync/sync_exist.test.js
@@ -36,8 +36,8 @@ describe('test/sync/sync_exist.test.js', function () {
    it('should sync first time ok', function *() {
      mm.data(npmService, 'getShort', ['byte']);
      mm.data(totalService, 'getTotalInfo', {last_exist_sync_time: 0});
-      var data = yield* sync();
-      data.successes.should.eql(['byte']);
+      var data = yield sync();
+      data.successes[0].should.equal('byte');
    });

    it('should sync common ok', function *() {
@@ -47,7 +47,7 @@ describe('test/sync/sync_exist.test.js', function () {
      });
      mm.data(totalService, 'getTotalInfo', {last_exist_sync_time: Date.now()});
      var data = yield* sync();
-      data.successes.should.eql(['byte']);
+      data.successes[0].should.equal('byte');

      mm.data(npmService, 'getAllSince', []);
      var data = yield* sync();
@@ -62,7 +62,7 @@ describe('test/sync/sync_exist.test.js', function () {
      ]);
      mm.data(totalService, 'getTotalInfo', {last_exist_sync_time: Date.now()});
      var data = yield* sync();
-      data.successes.should.eql(['byte']);
+      data.successes[0].should.equal('byte');

      mm.data(npmService, 'getAllSince', []);
      var data = yield* sync();
--- a/tools/resync_npm_issue_87.js
+++ b/tools/resync_npm_issue_87.js
@@ -0,0 +1,43 @@
+// try to fix https://github.com/npm/registry/issues/87
+
+const urllib = require('urllib');
+const co = require('co');
+
+const sleep = ms => cb => setTimeout(cb, ms);
+
+// node resync_npm.js [registry]
+const registry = process.argv[2] || require('../config').sourceNpmRegistry;
+const url = 'https://os.alipayobjects.com/rmsportal/eDMScnlBhNhHaGXMJWxjvCjfxMHhYwEx.html';
+
+co(function* () {
+  const result = yield urllib.request(url);
+  const items = result.data.toString().split('\n');
+  let count = 0;
+  for (let item of items) {
+    item = item.trim().split(',');
+    let name = item[0];
+    name = name.substring(1, name.length - 1);
+    if (!name) {
+      continue;
+    }
+
+    const r = yield urllib.request(`${registry}/${name}/sync`, {
+      method: 'PUT',
+      dataType: 'json',
+    });
+
+    r.data = r.data || {};
+
+    count++;
+    console.log('#%d %s %s, log: %s',
+      count, name, r.status, `${registry}/${name}/sync/log/${r.data.logId}`);
+    if (count % 50 === 0) {
+      yield sleep(10000);
+    }
+  }
+  console.log('All %d packages sync done', items.length);
+  process.exit(0);
+}).catch(err => {
+  console.error(err.stack);
+  process.exit(1);
+});
--- a/view/web/package.html
+++ b/view/web/package.html
@@ -15,13 +15,16 @@
      <% } %>
      <div class="pack-ver">
        <img title="<%= package.version %>" src="/badge/v/<%= package.name %>.svg?style=flat-square">
-        <% if (package.engines) {%>
+        <% if (package.engines) { %>
          <% for (var k in package.engines) {
            var engine = package.engines[k];
          %>
            <img title="<%= engine.title %>" src="<%- engine.badgeURL %>">
        <% }
-        }%>
+        } %>
+        <% if (package.snyk) { %>
+        <a class="badge-link" href="<%= package.snyk.url %>" target="_blank"><img title="Known Vulnerabilities" src="<%- package.snyk.badge %>"></a>
+        <% } %>
      </div>
      <div class="pack-repo">
        <% if (package.fromNow) { %>
@@ -52,14 +55,19 @@

        <% if (package.license || package.repository || package.bugs.url) {%> &middot; <% } %>

-        <% if (!package._publish_on_cnpm) { %>
+        <% if (!package.isPrivate) { %>
        <a href="https://www.npmjs.org/package/<%= package.name %>" target="_blank">Original npm</a>
        <% } %>
+
+        <% if (package.dist.tarball) { %>
+          &middot;
+          <a href="<%= package.dist.tarball %>" target="_blank">Tarball</a>
+        <% } %>
      </div>
      <div class="pack-install">
        <pre class="sh">$ <%- config.npmClientName %> install <%= package.name %> <% if (package.preferGlobal) { %>-g<% } %></pre>
        <div class="pack-sync">
-          <% if (package._publish_on_cnpm) { %>
+          <% if (package.isPrivate) { %>
            Private package
          <% } else { %>
            <a href="/sync/<%= package.name %>" target="_blank">SYNC</a> missed versions from
--- a/worker.js
+++ b/worker.js
@@ -1,18 +1,5 @@
-/**!
- * Copyright(c) cnpmjs.org and other contributors.
- * MIT Licensed
- *
- * Authors:
- *  dead_horse <dead_horse@qq.com>
- *  fengmk2 <fengmk2@gmail.com> (http://fengmk2.com)
- */
-
 'use strict';

-/**
- * Module dependencies.
- */
-
 var graceful = require('graceful');
 var registry = require('./servers/registry');
 var web = require('./servers/web');
Author	SHA1	Message	Date
fengmk2	373b535bc5	Release 2.19.4	2017-03-26 23:23:41 +08:00
fengmk2	2daa34faab	feat: need to sync sourceNpmRegistry also (#1153 ) make sure package data be update event replicate down. https://github.com/npm/registry/issues/129	2017-03-24 00:18:34 +08:00
fengmk2	10170537dd	docs: change user.json to utf8mb4	2017-03-09 10:29:57 +08:00
fengmk2	598dc08ef6	Release 2.19.3	2017-02-22 00:06:28 +08:00
fengmk2	f0238f6063	fix: should get package from orginal registry when package is unpublished (#1130 )	2017-02-22 00:05:29 +08:00
fengmk2	0d86d05d45	Release 2.19.2	2017-01-05 19:41:23 +08:00
fengmk2	1598c2a3d9	fix: should auto sync un-deprecate message (#1105 )	2017-01-05 19:40:36 +08:00
fengmk2	997e9c5cb5	Release 2.19.1	2016-12-29 00:54:44 +08:00
Yuwei Ba	9cfe0502dd	fix: try to use the best repository url (#1102 )	2016-12-29 00:53:54 +08:00
fengmk2	ff2536fcc5	Release 2.19.0	2016-12-21 00:26:09 +08:00
21paradox	df5564b1ea	feat: keyword search with limit to support keywords > 100 (#1097 ) max limit to 10000	2016-12-21 00:24:31 +08:00
fengmk2	093031e720	Release 2.18.0	2016-12-05 21:39:19 +08:00
fengmk2	65b595b899	fix: support downloads total on scope package (#1088 ) closes https://github.com/cnpm/cnpmjs.org/issues/1078	2016-12-05 21:38:38 +08:00
fengmk2	bf5f02908f	fix: try to sync from official replicate (#1076 )	2016-11-20 01:27:44 +08:00
alsotang	6c3140d422	feat: add change password script (#1070 )	2016-11-17 21:03:31 +08:00
fengmk2	dd27edc36b	test: skip always fail tests	2016-11-16 03:03:08 +08:00
fengmk2	2065c975e0	test: add node v7	2016-11-16 03:03:08 +08:00
fengmk2	0020a33e65	feat: show more sync info	2016-11-16 03:03:08 +08:00
fengmk2	dd423c08d8	Release 2.17.2	2016-11-13 23:26:39 +08:00
fengmk2	f9f049c177	fix: ignore long package name on unpublished sync (#1067 ) closes https://github.com/cnpm/cnpmjs.org/issues/1066	2016-11-13 23:23:59 +08:00
fengmk2	034bdbb344	Release 2.17.1	2016-11-08 17:26:45 +08:00
Yiyu He	d7d4cb7fa7	fix: add publish_time for private packages (#1061 )	2016-11-08 17:24:07 +08:00
fengmk2	b83483e3c0	Release 2.17.0	2016-11-03 00:02:38 +08:00
fengmk2	9b909b3233	feat: make snyk.io url configable (#1058 ) closes https://github.com/cnpm/mirrors/issues/116	2016-11-03 00:01:41 +08:00
fengmk2	fac9df1de5	Release 2.16.2	2016-09-27 18:16:42 +08:00
fengmk2	3262adadd4	fix: try to use config.registryHost first on setDownloadURL (#1044 )	2016-09-27 18:16:07 +08:00
fengmk2	74e9bdde72	Release 2.16.1	2016-08-22 21:51:23 +08:00
Yiyu He	f567ef5567	refactor: refine publishable's code (#1022 )	2016-08-22 21:48:47 +08:00
fengmk2	c89d7023c4	Release 2.16.0	2016-08-22 19:42:27 +08:00
Yiyu He	72aa64258f	feat: admin can do everything (#1021 )	2016-08-22 19:04:59 +08:00
dead_horse	b5b63c11f2	Release 2.15.0	2016-08-22 14:34:38 +08:00
fengmk2	4da684f3b6	feat: return dist-tag on package registry (#1020 )	2016-08-22 14:09:05 +08:00
Greenkeeper	fb29419c27	chore(package): update supertest to version 2.0.0 (#1004 ) https://greenkeeper.io/	2016-08-04 08:43:29 +08:00
fengmk2	8789dadaef	Release 2.14.0	2016-08-04 08:42:37 +08:00
wenbing	1c936d4b73	feat: password may contains ":" (#999 )	2016-08-04 08:40:12 +08:00
fengmk2	7c2f947518	fix: limit sync fails email notice (#1006 ) and skip JSONResponseFormatError and 504 server error	2016-08-02 21:42:31 +08:00
fengmk2	bfbfef87d2	Release 2.13.0	2016-07-26 18:33:32 +08:00
fengmk2	ea3beceef7	feat: enable maxrequests middleware (#1003 )	2016-07-26 18:32:33 +08:00
fengmk2	b0f0bc0331	Release 2.12.2	2016-07-11 12:48:36 +08:00
Yiyu He	932547af2a	fix: getModuleByRange don't list all packages (#990 )	2016-07-11 12:46:52 +08:00
fengmk2	79ac9fb823	fix: should show new version package count (#984 )	2016-07-04 00:06:42 +08:00
fengmk2	a437eb01eb	Release 2.12.1	2016-07-01 17:59:38 +08:00
fengmk2	6d921a9c46	fix: make sure chagnes stream destroy (#982 ) closes #981	2016-07-01 17:58:24 +08:00
Greenkeeper	abcb81d28c	chore(package): update semver to version 5.2.0 (#978 ) https://greenkeeper.io/	2016-07-01 15:27:10 +08:00
fengmk2	ea54f32805	deps: use ^ instead of ~ (#976 )	2016-06-27 01:01:23 +08:00
Greenkeeper	3dbf29afac	chore(package): update mini-logger to version 1.1.1 (#973 ) https://greenkeeper.io/	2016-06-26 13:32:53 +08:00
dead_horse	2c520d97a9	Release 2.12.0	2016-06-26 13:30:41 +08:00
fengmk2	38148b24cb	fix: logger seperator should be one EOL (#972 )	2016-06-26 13:28:42 +08:00
fengmk2	97895bdfa5	feat: add security check badge for public package (#971 )	2016-06-26 13:18:33 +08:00
dead_horse	0c1c12680c	Release 2.11.0	2016-06-25 21:06:08 +08:00
fengmk2	6485e4b019	feat: add changes stream syncer (#970 ) * feat: add changes stream syncer fix scope package missing problem close #908 * refactor: use let instead of var	2016-06-25 21:05:00 +08:00
Greenkeeper	ee1e084455	chore(package): update pg to version 5.1.0 (#953 ) https://greenkeeper.io/	2016-06-11 13:02:41 +08:00
dead_horse	5c8d679a6d	Release 2.10.1	2016-06-05 23:37:44 +08:00
fengmk2	94dc13088c	fix: should sync missing public scoped package on install (#946 ) * fix: should sync missing public scoped package on install closes #938 * refactor: sync all scoped packages when config.scopes is empty * test: test on node 6	2016-06-05 23:34:10 +08:00
fengmk2	9b572d77aa	docs: update npmjs.com, cnpmjs.org and npm.taobao.org relation (#945 )	2016-06-05 22:14:56 +08:00
Greenkeeper	c1fa6de038	chore(package): update bytes to version 2.4.0 (#943 ) https://greenkeeper.io/	2016-06-02 10:53:05 +08:00
alsotang	227e1927b4	userService (#926 )	2016-05-23 17:50:29 +08:00
Greenkeeper	ea98b61ddf	chore(package): update should to version 8.4.0 (#928 ) https://greenkeeper.io/	2016-05-23 10:27:09 +08:00
Greenkeeper	9d1d60b764	chore(package): update humanize-ms to version 1.2.0 (#927 ) https://greenkeeper.io/	2016-05-21 00:34:18 +08:00
Greenkeeper	7509d909e8	chore(package): update kcors to version 1.2.1 (#918 ) https://greenkeeper.io/	2016-05-14 21:24:17 +08:00
Greenkeeper	9612f64aba	chore(package): update urllib to version 2.9.0 (#898 ) https://greenkeeper.io/	2016-04-21 10:38:09 +08:00
fengmk2	1b04d59539	Release 2.10.0	2016-04-15 23:45:54 +08:00
fengmk2	e6998506f3	feat: show tarball url on package page (#894 ) closes #893	2016-04-15 23:44:07 +08:00
Greenkeeper	576397b2b0	chore(package): update koa-mock to version 1.6.1 (#891 ) http://greenkeeper.io/	2016-04-13 16:53:16 +08:00
Greenkeeper	6c1935bc61	chore(package): update koa-mock to version 1.6.0 (#890 ) http://greenkeeper.io/	2016-04-13 09:58:15 +08:00
fengmk2	1e68cfa568	Release 2.9.5	2016-04-12 11:24:10 +08:00
fengmk2	8188d9a07a	fix: change logo url to a better https source closes #887	2016-04-12 11:21:58 +08:00
Nimo Chu	d7ae8d28c1	fix: http://cnpmjs.org/package/fms pre style (#739 )	2016-04-09 12:38:30 +08:00
fengmk2	afa5639752	Release 2.9.4	2016-04-09 12:34:39 +08:00
fengmk2	e0213e18a0	fix: don't sync constructor package on exists mode (#883 ) closes #857	2016-04-09 12:33:52 +08:00
Greenkeeper	a74d60c7eb	Update utility to version 1.7.0 🚀 http://greenkeeper.io/	2016-04-07 09:22:29 +08:00
fengmk2	fcabc49d39	chore: update sponsor link	2016-04-05 23:06:48 +08:00
fengmk2	ad0722b57b	Release 2.9.3	2016-04-05 22:39:37 +08:00
fengmk2	6e11a2c8a7	fix: use better diff time to check sync status closes #858	2016-04-05 15:43:29 +08:00
Greenkeeper	0329cfad61	Update sequelize to version 3.21.0 🚀 http://greenkeeper.io/	2016-04-04 17:19:45 +08:00
Yiyu He	ede8a3ce5f	Merge pull request #876 from cnpm/greenkeeper-agentkeepalive-2.1.0 Update agentkeepalive to version 2.1.0 🚀	2016-04-02 17:49:39 +08:00
greenkeeperio-bot	cde4f2179e	chore(package): update agentkeepalive to version 2.1.0 http://greenkeeper.io/	2016-04-02 11:04:29 +08:00
Yiyu He	374dc17a6b	Merge pull request #875 from cnpm/greenkeeper-pg-4.5.2 Update pg to version 4.5.2 🚀	2016-03-31 01:41:35 +08:00
greenkeeperio-bot	626259c379	chore(package): update pg to version 4.5.2 http://greenkeeper.io/	2016-03-31 01:33:38 +08:00
fengmk2	36644bda67	Release 2.9.2	2016-03-29 17:56:05 +08:00
fengmk2	236e9e2fb3	Merge pull request #874 from dickeylth/master fix: override antd for ul & ol list number & icon.	2016-03-29 17:54:06 +08:00
dickeylth	b551216517	Update main.css override antd for ul & ol list number & icon.	2016-03-29 17:15:28 +08:00
dead_horse	c8d9d9f075	Release 2.9.1	2016-03-29 15:31:52 +08:00
Yiyu He	4d43588b10	Merge pull request #872 from cnpm/add-more-ua-info-on-syncer refactor: add more ua info on syncer	2016-03-29 15:31:04 +08:00
fengmk2	5f71add418	refactor: add more ua info on syncer	2016-03-29 14:53:05 +08:00