久久久久se色偷偷亚洲精品av,亚洲一级黄色视频,亚洲一区二区三区免费

MONGODB SPRING DISTINCT

paulwong — Tue, 18 Oct 2022 02:22:00 GMT

SPRING 框架�?如果要做去重�Q�在数据量大的时候会(x��)爆ERROR�Q�可改用如下写法�Q?br />

  private boolean needReorderCheck(String requestId) {
        boolean result = false;
//        try(MongoCursor mongoCursor =
//                mongoTemplate.getCollection(mongoTemplate.getCollectionName(AccountNumProductLineIndex.class))
//                             .distinct(KEY, Filters.eq(REQUEST_ID, requestId), String.class)
//                             .iterator()
//                )
        try(MongoCursor mongoCursor =
                mongoTemplate.getCollection(mongoTemplate.getCollectionName(AccountNumProductLineIndex.class))
                             .aggregate(
                                 Arrays.asList(
                                    Aggregates.project(
                                                    Projections.fields(
                                                                    Projections.excludeId(),
                                                                   Projections.include(KEY),
                                                                   Projections.include(REQUEST_ID)
                                                                )
                                               ),
                                    Aggregates.match(Filters.eq(REQUEST_ID, requestId)),
                                    Aggregates.group("$" + KEY)
                                 )
                              )
                             .allowDiskUse(true)
                             .iterator();
        )
        {
            String key = null;
            boolean breakMe = false;
            LOGGER.info("needReorderCheck.key --> start");
            while(mongoCursor.hasNext()) {
                if(breakMe) {
                    mongoCursor.close();
                    break;
                }
                Document keyDocument = mongoCursor.next();
                key = keyDocument.getString("_id");
//                key = mongoCursor.next().getString(KEY);
//                LOGGER.info("needReorderCheck.keyDocument --> {}, key --> {}", keyDocument, key);
                try(MongoCursor indexMongoCursor =
                        mongoTemplate.getCollection(AccountNumProductLineIndex.COLLECTION_NAME)
                                        .find(Filters.and(Filters.eq(REQUEST_ID, requestId), Filters.eq(KEY, key)))
                                        .iterator()
                )
                {
                    int preIndex = -1, currentIndex = -1;
                    Document preIndexDocument = null, currentIndexDocument;
                    while(indexMongoCursor.hasNext()) {
                        currentIndexDocument = indexMongoCursor.next();
//                        System.out.println(currentIndexDocument.toJson());
                        if(preIndexDocument != null) {
                             currentIndex = currentIndexDocument.getInteger(INDEX);
                             preIndex = preIndexDocument.getInteger(INDEX);
                             if(currentIndex - preIndex > 1) {
                                indexMongoCursor.close();
                                breakMe = true;
                                result = true;
                                break;
                            }
                        }
                        preIndexDocument = currentIndexDocument;
                    }
                }
            }
        }

        return result;
    }

paulwong 2022-10-18 10:22 发表评论

MONGODB SHELL

paulwong — Mon, 10 Jan 2022 03:10:00 GMT

mongo -u admin -p 123456 --authenticationDatabase admin
use admin
db.createUser({
user : "paul",
pwd : "123456",
roles : [{role : "readWrite", db : "batch"}]
})

#增加权限
db.grantRolesToUser(
  "paul",
  [
    { "role" : "dbOwner",
      "db" : "mcra"
    }
  ]
)

paulwong 2022-01-10 11:10 发表评论

WEBFLUX + SPRING SESSION + REACTIVE MONGODB

paulwong — Wed, 22 Dec 2021 01:24:00 GMT

��d��依赖�Q�pom.xml

<dependency>
   <groupId>org.springframework.bootgroupId>
   <artifactId>spring-boot-starter-webfluxartifactId>
dependency>



<dependency>
   <groupId>org.springframework.sessiongroupId>
   <artifactId>spring-session-data-mongodbartifactId>
dependency>

<dependency>
   <groupId>org.springframework.bootgroupId>
   <artifactId>spring-boot-starter-data-mongodb-reactiveartifactId>
dependency>

配置文�g�Q�application.yaml

spring:
   session:
      store-type: mongodb
      timeout: 30s
      mongodb:
         collection-name: WEB_SESSIONS

java配置�Q�HttpSessionConfiguration.java�Q?/h2>
package com.paul.testmicroservicecommon.config;

import org.springframework.boot.autoconfigure.session.MongoSessionProperties;
import org.springframework.boot.autoconfigure.session.SessionProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.session.config.ReactiveSessionRepositoryCustomizer;
import org.springframework.session.data.mongo.ReactiveMongoSessionRepository;
import org.springframework.session.data.mongo.config.annotation.web.reactive.EnableMongoWebSession;

@EnableMongoWebSession
@EnableConfigurationProperties(MongoSessionProperties.class)
public class HttpSessionConfiguration {

    @Bean
    public ReactiveSessionRepositoryCustomizer customize(
        SessionProperties sessionProperties,
        MongoSessionProperties mongoSessionProperties
    ){
        return c -> {
            c.setMaxInactiveIntervalInSeconds((int)sessionProperties.getTimeout().getSeconds());
            c.setCollectionName(mongoSessionProperties.getCollectionName());
        };
    }

}

paulwong 2021-12-22 09:24 发表评论

MONGODB聚合资源

paulwong — Thu, 08 Apr 2021 05:38:00 GMT

MongoDB Projections and Aggregations in Spring Boot
https://www.javaprogramto.com/2020/05/spring-boot-data-mongodb-projections-aggregations.html

In spring data mongodb how to achieve pagination for aggregation
https://stackoverflow.com/questions/34427241/in-spring-data-mongodb-how-to-achieve-pagination-for-aggregation

Create a ProjectionOperation with Difference with spring mongodb
https://stackoverflow.com/questions/46786577/create-a-projectionoperation-with-difference-with-spring-mongodb

paulwong 2021-04-08 13:38 发表评论

SPRING DATA MONGODB 教程

paulwong — Fri, 20 Nov 2020 06:45:00 GMT

https://github.com/eugenp/tutorials/tree/master/persistence-modules/spring-data-mongodb

paulwong 2020-11-20 14:45 发表评论

Finding slow queries in MongoDB

paulwong — Fri, 27 Mar 2020 15:35:00 GMT

Database Profiling

MongoDB Profiler is a db profiling system that can help identify inefficient

or slow queries and operations.

Levels of profiles available are:

Level	Setting
0	Off. & No profiling
1	On & only includes slow operations
2	On & Includes all operations

We can enable it by setting the Profile level value using the following
command in mongo shell :

"db.setProfilingLevel(1)"

By default, mongod records slow queries to its log, as defined by slowOpThresholdMs.

NOTE

Enabling database profiler puts negative impact on MongoDB’s performance.

It’s better to enable it for specific intervals & minimal on Production Servers.

We can enable profiling on a mongod basis but This setting will not propagate
across a replica set and sharded cluster.

We can view the output in the system.profile collection in mongo shell using show profile command, or using following:

db.system.profile.find( { millis : { $gt : 200 } } )

Command returns operations that took longer than 200 ms. Similarly we
can change the values as per our need.

Enabling profile for an entire mongod instance.

For the purpose of development in testing, we can enable database profiling/settings for an
entire mongod instance. The profiling level will be applied to all databases.

NOTE:

We can't enable the profiling settings on a mongos instance. To enable the profiling in

shard clusters, we have to enable/start profiling for each mongod instance in cluster.

Query for the recent 10 entries

db.system.profile.find().limit(10).sort( { ts : 1 } ).pretty()

Collection with the slowest queries(No. Of queries)

db.system.profile.group({key: {ns: true}, initial: {count: 0}, reduce: function(obj,prev){ prev.count++;}})

Collection with the slowest queries(No. Of millis spent)

db.system.profile.group({key: {ns: true}, initial: {millis: 0}, reduce: function(obj, prev){ prev.millis += obj.millis;}})

Most recent slow query

db.system.profile.find().sort({$natural: -1}).limit(1)

Single slowest query(Right now)

db.system.profile.find().sort({millis: -1}).limit(1)

paulwong 2020-03-27 23:35 发表评论

paulwong — Tue, 17 Mar 2020 01:39:00 GMT

正好做Mongodb��M��复制��试使用Spring Boot Data Mongodb Starter插�g链接讉K��Mongodb数据库集��?/p>

遇到的坑�Q?/p>

spring.data.mongodb.host和spring.data.mongodb.port形式不适合集群配置�Q�会(x��)报host无法识别异常
spring.data.mongodb.uri中经常抛出authentication failed异常

解决办法�Q?/p>

对于�W�一个坑�Q�请使用spring.data.mongodb.uri�?span style="background-color: yellow;">如果使用了uri�Q�则其余的host/username/password/db/auth-db�q�些全部无效�?/span>
对于�W�二个坑�Q�请在spring.data.mongodb.uri中指定replicaSet和authsource�Q�另外记得把所有集��节�Ҏ(gu��)��务器地址都列全�?br />如果auth-db和db是同一个，则无需加authsource�Q�如果不同，则加authsource=admin

我没有把authsource指定�Q�所以一直报authentication failed异常。然后只好一点点��d��掘问题点�Q�最后查到在com.mongodb.ConnectionString�c�M��的createCredentials�?br />

private MongoCredential createCredentials(final Map> optionsMap, final String userName,
                                              final char[] password) {
        AuthenticationMechanism mechanism = null;
        String authSource = (database == null) ? "admin" : database;
        String gssapiServiceName = null;
        String authMechanismProperties = null;

        for (final String key : AUTH_KEYS) {
            String value = getLastValue(optionsMap, key);

            if (value == null) {
                continue;
            }

            if (key.equals("authmechanism")) {
                mechanism = AuthenticationMechanism.fromMechanismName(value);
            } else if (key.equals("authsource")) {
                authSource = value;
            } else if (key.equals("gssapiservicename")) {
                gssapiServiceName = value;
            } else if (key.equals("authmechanismproperties")) {
                authMechanismProperties = value;
            }
        }

        MongoCredential credential = null;
        if (mechanism != null) {
            switch (mechanism) {
                case GSSAPI:
                    credential = MongoCredential.createGSSAPICredential(userName);
                    if (gssapiServiceName != null) {
                        credential = credential.withMechanismProperty("SERVICE_NAME", gssapiServiceName);
                    }
                    break;
                case PLAIN:
                    credential = MongoCredential.createPlainCredential(userName, authSource, password);
                    break;
                case MONGODB_CR:
                    credential = MongoCredential.createMongoCRCredential(userName, authSource, password);
                    break;
                case MONGODB_X509:
                    credential = MongoCredential.createMongoX509Credential(userName);
                    break;
                case SCRAM_SHA_1:
                    credential = MongoCredential.createScramSha1Credential(userName, authSource, password);
                    break;
                default:
                    throw new UnsupportedOperationException(format("The connection string contains an invalid authentication mechanism'. "
                                                                           + "'%s' is not a supported authentication mechanism",
                            mechanism));
            }
        } else if (userName != null) {
            credential = MongoCredential.createCredential(userName, authSource, password);
        }

        if (credential != null && authMechanismProperties != null) {
            for (String part : authMechanismProperties.split(",")) {
                String[] mechanismPropertyKeyValue = part.split(":");
                if (mechanismPropertyKeyValue.length != 2) {
                    throw new IllegalArgumentException(format("The connection string contains invalid authentication properties. "
                            + "'%s' is not a key value pair", part));
                }
                String key = mechanismPropertyKeyValue[0].trim().toLowerCase();
                String value = mechanismPropertyKeyValue[1].trim();
                if (key.equals("canonicalize_host_name")) {
                    credential = credential.withMechanismProperty(key, Boolean.valueOf(value));
                } else {
                    credential = credential.withMechanismProperty(key, value);
                }
            }
        }
        return credential;
    }

authSource默认�?x��)指向我们目标数据的数据库。然而在�w�䆾验证机制中我们通常需要指向admin。（非常��x��_�口�Q�代码作者在�q�里脑袋被men挤了么）。所以需要强制指定authSource中指定。具体指定方式如下：(x��)

mongodb://{用户名}:{密码}@{host1}:27017,{host2}:27017,{host3}:27017/{目标数据库}?replicaSet={复制集名�U�}&write=1&readPreference=primary&authsource={授权数据库}

paulwong 2020-03-17 09:39 发表评论

MongDB�q�接池参数serverSelectionTimeout、connectTimeout、maxWaitTime和socketTimeout介绍

paulwong — Sat, 07 Mar 2020 12:58:00 GMT

MongDB Client��h��查询数据�Q�需要包括五个阶�D�：(x��)

MongoDB Client需要找到可用的MongoDB Server

MongoDB Client需要和MongoDB Server建立�Q�new�Q�Connection

应用�E�序处理�U�程从Connection Pool中获取Connection

数据传输�Q�获取连接后�Q�进行Socket通信�Q�获取数据）

断开Collection

那么�Q�MongoDB Client驱动讄��中网�l�相关等待超时参数serverSelectionTimeout、connectTimeout、maxWaitTime和socketTimeout分别对应上面哪个环节呢？

参数serverSelectionTimeout�Q�对应第1个环节，即MongoDB Client需要找到可用的MongoDB Server所需要的�{�待旉��Q?nbsp; MongDB部��v的生产一般由多个服务器组成，要么作�ؓ(f��)一个复刉��或者作��Z��个分片集��，参数 serverSelectionTimeout的值即为多长时间内找不到合适服务器时候就军_��攑ּ�的时间间隔；

参数connectTimeout�Q�对应第2个环节，每次创徏Connection�Q�对应的�|�络�{�待。单位毫�U�数, 0表示没有限制�Q?/div>

参数maxWaitTime�Q�对应第3个环节，应用�E�序处理�U�程从连接池中获取Collection�Q�对应的�|�络�{�待旉��。单位毫�U�数�Q?表示不等待，负数表示�{�待旉��不确定；

参数socketTimeout�Q�对应第4个环节，获取Connection后，��有了Socket通信�Q�获取数据，卛_��MonogoDB Client 和MonogoDB Server的Socket通信�q�程中的�|�络�{�待旉��。单位毫�U�数�Q�默认配�|��ؓ(f��)0�Q�也��是没有限制�Q?nbsp; 没有��?旉��Ӟ��pȝ��Z��问题也不�Ҏ(gu��)��发现�Q�应该根据实际情况，�l�出合理的超时时间�?/div>

其他相关参数如下�Q?/div>

connectionsPerHost�Q�对mongo实例来说�Q�每个host允许链接的最大链接数,�q�些链接�I�闲时会(x��)攑օ�池中,如果链接被耗尽�Q��Q何请求链接的操作�?x��)被��d��{�待链接可用,推荐配置10

minPoolsSize�Q�当Connection�I�闲�?Connection Pool中最��Connection保有量；

threadsAllowedToBlockForConnectionMultiplier�Q�每个Connection的可以阻塞等待的�U�程队列敎ͼ�它以上面connectionsPerHost值相乘的�l�果��是��d��{�待的线�E�队列最大倹{��如果连接线�E�排满了队列��׃��(x��)抛出“Out of semaphores to get db”错误�?/div>

socketKeepAlive:该标志用于控制socket保持�z�d��的功能，通过防火墙保持连接活着

socketKeepAlive=false

autoConnectRetry�Q�这个控制是否在一个Connection�Ӟ��pȝ��?x��)自动重�?/div>

#true:假如Connection不能建立�?驱动��重试相同的server,有最大的重试�ơ数,默认�?5��?�q�样可以避免一些server因�ؓ(f��)一些阻塞操作零时down而驱动抛出异�?�q�个对��^滑过度到一个新的master,也是很有用的,注意:当集��ؓ(f��)复制集时,驱动��在�q�段旉��?��试链接到旧的master�?而不�?x��)马上链接到新master�?/div>

#false 当在�q�行socket��d��?不会(x��)��L��异常抛出,驱动已经有自动重建破坏链接和重试��L��? 推荐配置false

autoConnectRetry=false

#重新打开链接到相同server的最大毫�U�数,推荐配置�?�Q�如�?autoConnectRetry=true,表示旉��?5s

#com.jd.mongodbclient2.mongo.JDClientMongo.maxAutoConnectRetryTime=false

#表示当没有手动关闭游标时,是否有一个自动释放游标对象的�Ҏ(gu��)��,如果你��L��很小心的关闭游标,则可以将其设为false 推荐配置true

https://docs.mongodb.com/manual/reference/connection-string/#connections-connection-options

————————————————

原文链接�Q�https://blog.csdn.net/pursuer211/article/details/82994027

paulwong 2020-03-07 20:58 发表评论

paulwong — Fri, 21 Feb 2020 15:10:00 GMT

var bookIds = db.likes.find({userId:100}).map(function(like) {
return like.bookId;
});
var books = db.books.find({_id:{$in:bookIds}});

db.REPORT_ITEM.count({REQUEST_ID : db.BATCH_CONTROL.find({ FILE_NAME : "20200218_100000.file" }).map(function(like) {
return like._id;
})[0].str, JOB_TYPE_ENUM:"CHECK"})

paulwong 2020-02-21 23:10 发表评论

mongodb错误记录

paulwong — Fri, 21 Feb 2020 00:50:00 GMT

https://blog.csdn.net/wangxiaotongfan/article/details/81560463

paulwong 2020-02-21 08:50 发表评论

MONGODB安装

paulwong — Fri, 15 Nov 2019 09:30:00 GMT

https://www.jianshu.com/p/d3b31b7aa182

后箋初始化用户及(qi��ng)数据�?br />http://www.qianduan8.com/1786.html
https://zocada.com/setting-mongodb-users-beginners-guide/

如果要以认证的方式登录，需加以下内容至/etc/mongod.conf

security:
authorization: enabled

如果用GUI�q�接数据库时�Q�不昄��数据库列表，要加权限�Q�listDatabases
https://stackoverflow.com/questions/19458524/mongodb-show-dbs-and-show-log-without-clusteradmin-role

paulwong 2019-11-15 17:30 发表评论

MONGODB拓展操作

paulwong — Tue, 03 Sep 2019 07:52:00 GMT

MongoTemplate使用Cursor处理大数量的数据
https://blog.csdn.net/ClementAD/article/details/55210973

Spring Data MongoDB�p�d��之三�Q�数据库扚w��操作
https://blog.csdn.net/sinat_24044957/article/details/80646292

Distinct in Spring Data MongoDB
https://stackoverflow.com/questions/19203724/distinct-in-spring-data-mongodb

MONGODB　SQL语句
http://www.runoob.com/mongodb/mongodb-indexing.html

paulwong 2019-09-03 15:52 发表评论

Multiple MongoDB connectors with Spring Boot

paulwong — Thu, 20 Jun 2019 07:12:00 GMT

http://blog.marcosbarbero.com/multiple-mongodb-connectors-in-spring-boot/

https://github.com/yinjihuan/spring-boot-starter-mongodb-pool

https://github.com/lish1le/mongodb-plus

paulwong 2019-06-20 15:12 发表评论

SpringBoot使用MongoDB异常问题

paulwong — Wed, 29 May 2019 08:58:00 GMT

https://www.cnblogs.com/linzhanfly/p/9674778.html

paulwong 2019-05-29 16:58 发表评论

MONGODB去除_class字段

paulwong — Wed, 29 May 2019 06:18:00 GMT

加上此配�|�：(x��)

MongodbConfiguration.java

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.MongoDbFactory;
import org.springframework.data.mongodb.core.convert.DbRefResolver;
import org.springframework.data.mongodb.core.convert.DefaultDbRefResolver;
import org.springframework.data.mongodb.core.convert.DefaultMongoTypeMapper;
import org.springframework.data.mongodb.core.convert.MappingMongoConverter;
import org.springframework.data.mongodb.core.mapping.MongoMappingContext;

@Configuration
public class AppMongoConfig {
  @Autowired private MongoDbFactory mongoDbFactory;

  @Autowired private MongoMappingContext mongoMappingContext;

  @Bean
  public MappingMongoConverter mappingMongoConverter() {

    DbRefResolver dbRefResolver = new DefaultDbRefResolver(mongoDbFactory);
    MappingMongoConverter converter = new MappingMongoConverter(dbRefResolver, mongoMappingContext);
    converter.setTypeMapper(new DefaultMongoTypeMapper(null));

    return converter;
  }
}

paulwong 2019-05-29 14:18 发表评论

MongoDB健壮集群——用副本集做分片

paulwong — Fri, 18 Dec 2015 06:03:00 GMT

摘要: 1. MongoDB分片+副本集健壮的集群�Ҏ(gu��)��多个配置服务�?多个mongos服务�?nbsp; 每个片都是副本集正确讄��w架构图说明：(x��)1. 此实验环境在一台机器上通过不同port和dbpath实现启动不同的mongod实例2. �ȝ��9个mongod实例�Q�分别做成shard... 阅读全文

paulwong 2015-12-18 14:03 发表评论

paulwong — Fri, 18 Dec 2015 05:54:00 GMT

摘要: 参考资�?nbsp;reference: http://mongodb.blog.51cto.com/1071559/740131 http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/#sharding-setup-shard-collection感谢�|�友Mr.Sharp�Q�他�l�了我很�?.. 阅读全文

paulwong 2015-12-18 13:54 发表评论

MONGODB的复制与分片

paulwong — Fri, 18 Dec 2015 05:21:00 GMT

复制�Q��ؓ(f��)了防止单�Ҏ(gu��)��障，�?x��)有几个实例在运行，保持相同的数据�?br />

一般主从：(x��)一��d��从，��M��d��数据�Q�从作从��d��份数据用�Q�如果主宕机�Q�则整个MONGODB无法工作�?/li>

复制式主从：(x��)一动态主多从�Q�主由选�D产生�Q�当中一个主宕机�Q�其他的从会(x��)选出一个主�?/li>

适用场景�Q�高负荷的读多写��?br />
分片�Q�SHARDING�Q�一般数据库中的分库分表�Q�一个表分成几个表用。每个片再做复制�?br />
适用场景�Q�高负荷的写多读��。即如果发现MONGODB写不能支撑了�Q�则要�{此模式�?br />
安装配置服务器，安装ROUTER�Q�MONGOS�Q�安装分片服务器�Q�通知MONGOS挂蝲SHARD�?br />
如果只启用数据库的分片，则不同的表放在不同的分片上，即一个表只占一个分片，另一个表占另一个分片，如果做了表的分片�Q�则此表�?x��)分布在所有分片上�?br />

paulwong 2015-12-18 13:21 发表评论

mongodb的监控与性能优化

paulwong — Wed, 16 Dec 2015 10:50:00 GMT

摘要: .mongodb的监�?nbsp;mongodb可以通过profile来监控数据，�q�行优化。查看当前是否开启profile功能用命令db.getProfilingLevel() �q�回level�{��Q��gؓ(f��)0|1|2�Q�分别代表意思：(x��)0代表关闭�Q?代表记录慢命令，2代表全部开始profile功能为db.setProfilingLevel(level); #level�{��Q�值同上l... 阅读全文

paulwong 2015-12-16 18:50 发表评论

paulwong — Fri, 11 Dec 2015 07:03:00 GMT

MONGODB中，�׃��删除大量记录�?x��)十分耗时�Q�一般推荐由MONGODB自己在后台处理，只需在某个字�D�设一个烦(ch��)引的标签卛_��?br />

@Indexed(expireAfterSeconds=180)
private Date deletedAt;

以上代码�Q�如果字�D?/span>deletedAt有��|��那么��在180�U�后被MONGODB删除�Q�如果没��g��?x��)被删除�?/span>

扚w��新增�Q�小扚w��更新�Q�防止读取超�?br />

private  void insertAll(List list) {
        if (null != list) {
            int total = list.size();
            int count = (total + 50000 -1) / 50000;
            for (int i = 0; i < count; i++) {
                int toIndex = ((i +1) * 50000 > total) ? total : ((i +1) * 50000);
                log.info("toIndex = " + toIndex);
                mongoTemplate1.insertAll(list.subList(i * 50000, toIndex));
            }
        }
    }

扚w��更改

import java.util.Date;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.Query;
import org.springframework.data.mongodb.core.query.Update;

import com.tcl.project7.boss.gameapplication.yearendactivities.bigwheelgame.valueobject.SingleUseRedeemCode;

public class SingleUseRedeemCodeRepositoryImpl implements SingleUseRedeemCodeRepositoryCustom{

    @Autowired
    private MongoTemplate mongoTemplate1;

    public void batchUpdateSingleUseRedeemCodeList(String bigWheelGameAwardId) {

        Query query = new Query();
        query.addCriteria(Criteria.where("bigwheelgameawardid").is(bigWheelGameAwardId));
        mongoTemplate1.updateMulti(
                                    query,
                                    new Update().set("bigwheelgameawardid", "-1")
                                        .set("deletedat", new Date()),
                                    SingleUseRedeemCode.class);
    }

}

Expire Data from Collections by Setting TTL¶

New in version 2.2.

This document provides an introduction to MongoDB’s “time to live” or TTL collection feature. TTL collections make it possible to store data in MongoDB and have the mongod automatically remove data after a specified number of seconds or at a specific clock time.

Data expiration is useful for some classes of information, including machine generated event data, logs, and session information that only need to persist for a limited period of time.

A special TTL index property supports the implementation of TTL collections. The TTL feature relies on a background thread in mongod that reads the date-typed values in the index and removes expired documentsfrom the collection.

Procedures

To create a TTL index, use the db.collection.createIndex() method with theexpireAfterSeconds option on a field whose value is either a date or an array that contains date values.

NOTE

The TTL index is a single field index. Compound indexes do not support the TTL property. For more information on TTL indexes, see TTL Indexes.

Expire Documents after a Specified Number of Seconds

To expire data after a specified number of seconds has passed since the indexed field, create a TTL index on a field that holds values of BSON date type or an array of BSON date-typed objects and specify a positive non-zero value in the expireAfterSeconds field. A document will expire when the number of seconds in the expireAfterSeconds field has passed since the time specified in its indexed field. [1]

For example, the following operation creates an index on the log_events collection’s createdAt field and specifies the expireAfterSeconds value of 3600 to set the expiration time to be one hour after the time specified by createdAt.

db.log_events.createIndex( { "createdAt": 1 }, { expireAfterSeconds: 3600 } )

When adding documents to the log_events collection, set the createdAt field to the current time:

db.log_events.insert( {    "createdAt": new Date(),    "logEvent": 2,    "logMessage": "Success!" } )

MongoDB will automatically delete documents from the log_events collection when the document’screatedAt value [1] is older than the number of seconds specified in expireAfterSeconds.

[1]	(1, 2) If the field contains an array of BSON date-typed objects, data expires if at least one of BSON date-typed object is older than the number of seconds specified in `expireAfterSeconds`.

Expire Documents at a Specific Clock Time

To expire documents at a specific clock time, begin by creating a TTL index on a field that holds values of BSON date type or an array of BSON date-typed objects and specify an expireAfterSeconds value of0. For each document in the collection, set the indexed date field to a value corresponding to the time the document should expire. If the indexed date field contains a date in the past, MongoDB considers the document expired.

For example, the following operation creates an index on the log_events collection’s expireAt field and specifies the expireAfterSeconds value of 0:

db.log_events.createIndex( { "expireAt": 1 }, { expireAfterSeconds: 0 } )

For each document, set the value of expireAt to correspond to the time the document should expire. For instance, the following insert() operation adds a document that should expire at July 22, 201314:00:00.

db.log_events.insert( {    "expireAt": new Date('July 22, 2013 14:00:00'),    "logEvent": 2,    "logMessage": "Success!" } )

MongoDB will automatically delete documents from the log_events collection when the documents’expireAt value is older than the number of seconds specified in expireAfterSeconds, i.e. 0 seconds older in this case. As such, the data expires at the specified expireAt value.

paulwong 2015-12-11 15:03 发表评论

paulwong — Thu, 10 Dec 2015 12:09:00 GMT

We have a db collection that is around 30 million documents, and I need to trim it down, to only keeping the documents created on the last month.

One approach would be use the remove command with a condition on the created_at field (the collection already have an index on this field):

db.my_collection.remove({created_at: {$lte: new Date("11/01/2012")}});

But this approach will be very slow, instead of that, a better way to do it is rename the current collection (for instance to “old_collection”) using renameCollection. Then performing a query-and-insert from the “old_collection” into “my_collection”:

db.my_collection.renameCollection("old_collection");  
db.createCollection("my_collection");  
db.my_collection.createIndex(...); // recreate the indexes for the collection  
// copy docs from old collection into the new collection 
db.old_collection.find(     
{created_at: {$gte: new Date("11/01/2012")}}   ).sort({_id: -1}).forEach(     
function(row) { db.my_collection.insert(row); }   );  // drop old collection db.old_collection.drop();

This approach is typically faster than running a bunch of removes on your data

paulwong 2015-12-10 20:09 发表评论

MongoDB 固定集合(Capped Collections)

paulwong — Wed, 09 Dec 2015 06:41:00 GMT

MongoDB 固定集合�Q�Capped Collections�Q�是性能��且有着固定大小的集合，对于大小固定�Q�我们可以想象其��像一个环形队列，当集合空间用完后�Q�再插入的元素就�?x��)覆盖最初始的头部的元素�Q?/p>

创徏固定集合

我们通过createCollection来创��Z��个固定集合，且capped选项讄��为true�Q?/p>

>db.createCollection("cappedLogCollection",{capped:true,size:10000})

�q�可以指定文档个�?加上max:1000属性：(x��)

>db.createCollection("cappedLogCollection",{capped:true,size:10000,max:1000})

判断集合是否为固定集�?

>db.cappedLogCollection.isCapped()

如果需要将已存在的集合转换为固定集合可以��用以下命令：(x��)

>db.runCommand({"convertToCapped":"posts",size:10000})

以上代码��我们已存在�?posts 集合转换为固定集合�?/p>

固定集合查询

固定集合文档按照插入��序储存�?默认情况下查询就是按照插入顺序返回的,也可以��?natural调整�q�回��序�?/p>

>db.cappedLogCollection.find().sort({$natural:-1})

固定集合的功能特�?/h2>
可以插入�?qi��ng)更�?但更��C��能超出collection的大��?否则更新��p�|,不允许删�?但是可以调用drop()删除集合中的所有行,但是drop后需要显式地重徏集合�?/p>
�?2位机子上一个cappped collection的最大值约�?82.5M,64位上只受�pȝ��文�g大小的限制�?/p>

固定集合属性及(qi��ng)用法

属�?/h3>
属�?:对固定集合进行插入速度极快
属�?:按照插入��序的查询输出速度极快
属�?:能够在插入最新数据时,淘汰最早的数据

用法

用法1:储存日志信息
用法2:�~�存一些少量的文档

paulwong 2015-12-09 14:41 发表评论

MongoDB 聚合

paulwong — Tue, 08 Dec 2015 02:44:00 GMT

http://www.runoob.com/mongodb/mongodb-aggregate.html
MongoDB中聚�?aggregate)主要用于处理数据(诸如�l�计�q�_��?求和�{?�Q��ƈ�q�回计算后的数据�l�果。有点类似sql语句中的 count(*)�?/p>

aggregate() �Ҏ(gu��)��

MongoDB中聚合的�Ҏ(gu��)��使用aggregate()�?/p>

语法

aggregate() �Ҏ(gu��)��的基本语法格式如下所�C�：(x��)

>db.COLLECTION_NAME.aggregate(AGGREGATE_OPERATION)

实例

集合中的数据如下�Q?/p>

{    _id: ObjectId(7df78ad8902c)    title: 'MongoDB Overview',     description: 'MongoDB is no sql database',    by_user: 'w3cschool.cc',    url: 'http://www.w3cschool.cc',    tags: ['mongodb', 'database', 'NoSQL'],    likes: 100 }, {    _id: ObjectId(7df78ad8902d)    title: 'NoSQL Overview',     description: 'No sql database is very fast',    by_user: 'w3cschool.cc',    url: 'http://www.w3cschool.cc',    tags: ['mongodb', 'database', 'NoSQL'],    likes: 10 }, {    _id: ObjectId(7df78ad8902e)    title: 'Neo4j Overview',     description: 'Neo4j is no sql database',    by_user: 'Neo4j',    url: 'http://www.neo4j.com',    tags: ['neo4j', 'database', 'NoSQL'],    likes: 750 },

现在我们通过以上集合计算每个作者所写的文章敎ͼ�使用aggregate()计算�l�果如下�Q?/p>

> db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$sum : 1}}}]) { "result" : [ { "_id" : "w3cschool.cc", "num_tutorial" : 2 }, { "_id" : "Neo4j", "num_tutorial" : 1 } ], "ok" : 1 } >

以上实例�c�M��sql语句�Q?em style="border: 0px; margin: 0px; padding: 0px;"> select by_user, count(*) from mycol group by by_user

在上面的例子中，我们通过字段by_user字段�Ҏ(gu��)��据进行分�l�，�q�计��by_user字段相同值的��d��?/p>

下表展示了一些聚合的表达�?

表达�?/th>	描述	实例
$sum	计算��d��?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$sum : "$likes"}}}])
$avg	计算�q�_��?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$avg : "$likes"}}}])
$min	获取集合中所有文档对应值得最��倹{�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$min : "$likes"}}}])
$max	获取集合中所有文档对应值得最大倹{�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$max : "$likes"}}}])
$push	在结果文档中插入值到一个数�l�中�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", url : {$push: "$url"}}}])
$addToSet	在结果文档中插入值到一个数�l�中�Q�但不创建副本�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", url : {$addToSet : "$url"}}}])
$first	�Ҏ(gu��)��资源文档的排序获取第一个文档数据�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", first_url : {$first : "$url"}}}])
$last	�Ҏ(gu��)��资源文档的排序获取最后一个文档数�?/td>	db.mycol.aggregate([{$group : {_id : "$by_user", last_url : {$last : "$url"}}}])

��道的概�?/h2>
��道在Unix和Linux中一般用于将当前命��o(h��)的输出结果作��Z��一个命令的参数�?/p>
MongoDB的聚合管道将MongoDB文档在一个管道处理完毕后��结果传递给下一个管道处理。管道操作是可以重复的�?/p>
表达式：(x��)处理输入文档�q�输出。表辑ּ�是无状态的�Q�只能用于计��当前聚合管道的文档�Q�不能处理其它的文档�?/p>
�q�里我们介绍一下聚合框架中常用的几个操作：(x��)
$project�Q�修改输入文档的�l�构。可以用来重命名、增加或删除域，也可以用于创��结果以�?qi��ng)嵌套文档�?/li>
$match�Q�用于过滤数据，只输出符合条件的文档�?match使用MongoDB的标准查询操作�?/li>
$limit�Q�用来限制MongoDB聚合��道�q�回的文档数�?/li>
$skip�Q�在聚合��道中蟩�q�指定数量的文档�Q��ƈ�q�回余下的文档�?/li>
$unwind�Q�将文档中的某一个数�l�类型字�D�|��分成多条�Q�每条包含数�l�中的一个倹{�?/li>
$group�Q�将集合中的文档分组�Q�可用于�l�计�l�果�?/li>
$sort�Q�将输入文档排序后输出�?/li>
$geoNear�Q�输出接�q�某一地理位置的有序文档�?/li>

��道操作�W�实�?/h3>

1�?project实例

db.article.aggregate( { $project : {         title : 1 ,         author : 1 , }} );

�q�样的话�l�果中就只还有_id,tilte和author三个字段了，默认情况下_id字段是被包含的，如果要想不包含_id话可以这�?

db.article.aggregate( { $project : {         _id : 0 ,         title : 1 ,         author : 1 }});

2.$match实例

db.articles.aggregate( [ { $match : { score : { $gt : 70, $lte : 90 } } }, { $group: { _id: null, count: { $sum: 1 } } } ] );

$match用于获取分数大于70��于或等�?0记录�Q�然后将�W�合条�g的记录送到下一阶段$group��道操作�W�进行处理�?/p>

3.$skip实例

db.article.aggregate( { $skip : 5 });

�l�过$skip��道操作�W�处理后�Q�前五个文档�?�q��o(h��)"掉�?/p>

paulwong 2015-12-08 10:44 发表评论

mongodb�q�维之副本集实践

paulwong — Tue, 30 Jun 2015 02:30:00 GMT

摘要: 正式环境�Q?台机�?一台定时�Q务的机器�?服务器是阉K��云的ECS�Q?负蝲均衡用的是阿里云的SLB�Q?mysql用阿里云的RDS�Q?�~�存用阿里云的OCS�Q?�q�维基本上是都不需要担心了�Q?现在的云服务已经非常完善了，其实我们用阿里云的服务非常多�Q?大概�?0多个�c�d��的服务，感谢阉K��云�?而我的技术栈是nodejs + mongodb�Q�而阿里云有k-v兼容redis协议的nosql�Q�无mongodb... 阅读全文

paulwong 2015-06-30 10:30 发表评论

MONGODB优化资源

paulwong — Tue, 05 May 2015 09:32:00 GMT

把MongoDB当成�U�内存数据库使用
http://database.51cto.com/art/201309/411076.htm

Do you want a timeout?
http://blog.mongolab.com/2013/10/do-you-want-a-timeout/

MongoDB 查询��时异常的原因及(qi��ng)解决办法
http://database.51cto.com/art/201503/467581.htm

Mongo�q�接池操作MongoOptions
http://dawn-sky.iteye.com/blog/1343659

paulwong 2015-05-05 17:32 发表评论

MONGODB常用命��o(h��)

paulwong — Wed, 15 Apr 2015 09:39:00 GMT

启动�Q?/span>
mongod --config /usr/local/mongodb/conf/cms.conf

关闭�Q?/span>

ps -ef | grep mongo (查PID)
kill -15 pid

修复�Q?/span>

rm /data/mongodb/mongod.lock

mongod --dbpath /data/mongodb --repair

mongod --config /usr/local/mongodb/conf/cms.conf

�q�入命��o(h��)模式�Q?/span>

mongo admin -u root -p cloudDB

查看服务器连接数�Q?/span>

db.serverStatus().connections

查看版本�Q?/span>

db.runCommand({"buildInfo":1})

查看�l�计�Q?/span>

mongostat -u root -p cloudDB

备䆾数据�Q?/span>

mongodump -u cms -p cms -d cms -o /data/dump/cms

恢复数据�Q?/span>

mongorestore -u cms -p cms -d cms /data/dump/cms

paulwong 2015-04-15 17:39 发表评论

几大NOSQL数据库性能比较

paulwong — Thu, 29 Jan 2015 16:16:00 GMT

Apache Cassandra NoSQL Performance Benchmarks
http://planetcassandra.org/nosql-performance-benchmarks

NoSQL Benchmarking
http://www.cubrid.org/blog/dev-platform/nosql-benchmarking/

http://www.badrit.com/blog/2013/11/18/redis-vs-mongodb-performance#.VMpfW2SUfsg

How many requests per second can I get out of Redis?
http://skipperkongen.dk/2013/08/27/how-many-requests-per-second-can-i-get-out-of-redis/

redis性能��试�Q�测试�ƈ发性能
http://my.oschina.net/pblack/blog/102394

paulwong 2015-01-30 00:16 发表评论

paulwong — Tue, 06 Jan 2015 15:39:00 GMT

上一��：(x��)Mongodb VS Mysql 查询性能�Q�测试了 mongodb �?mysql 的查询性能。结果说�?mongodb 性能可以�Q?可以代替 mysql 来��用�?br />
但是�q�个��试都是在百万��别，我的场景�?KW �U�别。所以还要对 mongodb �?kw �U�别下测试效果�?br />
我测试环境是 4G 内存�Q�有好些内存被其它程序占用）�Q?kw 数据�Q�查询随机生�?id�Q�一�ơ查�?20 个id�Q��?br />
在这��L(f��ng)��环境中测试不理想�Q�比较失望。��^均一�ơ查�?500ms�Q�比 mysql �q�差�Q�特别是在�ƈ发查询下�Q�性能较差。很底的吞吐量）。查看其索引大小�Q�用 db.mycoll.stats() 可以查询�Q�：(x��)2kw 数据中有 1.1G 左右的烦(ch��)引，存储的数据在 11G 左右�?br />
��试�q�程中发�?iowait �?50% 左右�Q�看来还�?io 的瓶颈。还看到 mongodb 使用的内存不多（��于索引的大��，看来�q�机器不��_��来测试）�?br />
换了个有可用 6G 内存的机器。在 50 个�ƈ发下�Q�可以达到��^�?100 ms 左右�Q�算比较满意�Q�但是�ƈ发好像能力不够强。但�q�个性能不能由我控制�Q�还由机器的可用内存控制。原因就�?mongodb 没有指定可占用的内存大小�Q�它把所有空闲内存当�~�存使用�Q�既是优点也是缺点：(x��)优点--可以最大限度提升性能�Q�缺�?-�Ҏ(gu��)��受其它程序干扎ͼ�占用了它的缓存）。由我测试来看，它抢占内存的能力不强。mongodb 是用内存映射文�g vmm�Q�官方的说明�Q?br />
Memory Mapped Storage Engine

This is the current storage engine for MongoDB, and it uses memory-mapped files for all disk I/O. Using this strategy, the operating system's virtual memory manager is in charge of caching. This has several implications:

There is no redundancy between file system cache and database cache: they are one and the same.
MongoDB can use all free memory on the server for cache space automatically without any configuration of a cache size.
Virtual memory size and resident size will appear to be very large for the mongod process. This is benign: virtual memory space will be just larger than the size of the datafiles open and mapped; resident size will vary depending on the amount of memory not used by other processes on the machine.
Caching behavior (such as LRU'ing out of pages, and laziness of page writes) is controlled by the operating system: quality of the VMM implementation will vary by OS.
所以这么来看，我觉�?mongodb 没有指定内存大小来保证正常的�~�存是个�~�点。应该至��保证烦(ch��)引全部能攑ֈ�内存中。但�q�个行�ؓ(f��)不是由启动程序决定，而是��q��境决定（��中不��Q��?br />
官方也有�D�内容说到烦(ch��)引放到内存中�Q?br />
If your queries seem sluggish, you should verify that your indexes are small enough to fit in RAM. For instance, if you're running on 4GB RAM and you have 3GB of indexes, then your indexes probably aren't fitting in RAM. You may need to add RAM and/or verify that all the indexes you've created are actually being used.

�q�是希望 mongodb 中可以指定内存大��，��保它有��_��内存加蝲索引�?br />
��结�Q�大数据量下�Q�kw�U�）mongodb �q�发查询不够理想�Q?00-200/s�Q�。写数据很快�Q�我的环境，�q�程提交�q?1w/s�Q�估计达�?1.5W/s 是没问题的，基本不受大数据量的媄(ji��ng)响）�?br />
贴个��试数据�Q?br />
1 id(内存使用 <1.5g) 10 id(内存使用 2-3g) 20 id(内存使用 >4g)
1 2 3 1 2 3 1 2 3
total time 17.136 25.508 17.387 37.138 33.788 25.143 44.75 31.167 30.678
1 thread thruput 583.5668 392.0339 575.1423 269.266 295.9631 397.725 223.4637 320.8522 325.9665
total time 24.405 22.664 24.115 41.454 41.889 39.749 56.138 53.713 54.666
5 thread thruput 2048.76 2206.142 2073.398 1206.156 1193.631 1257.893 890.6623 930.8733 914.6453
total time 27.567 26.867 28.349 55.672 54.347 50.93 72.978 81.857 75.925
10 thread thruput 3627.526 3722.038 3527.461 1796.235 1840.028 1963.479 1370.276 1221.643 1317.089
total time 51.397 57.446 53.81 119.386 118.015 76.405 188.962 188.034 138.839
20 thread thruput 3891.278 3481.53 3716.781 1675.238 1694.7 2617.63 1058.414 1063.637 1440.517
total time 160.038 160.808 160.346 343.559 352.732 460.678 610.907 609.986 1411.306
50 thread thruput 3124.258 3109.298 3118.257 1455.354 1417.507 1085.357 818.4552 819.6909 354.2818
total time 2165.408 635.887 592.958 1090.264 1034.057 1060.266 1432.296 1466.971 1475.061
100 thread thruput 461.8067 1572.606 1686.46 917.209 967.0647 943.1595 698.1797 681.6767 677.9381
上面的测试分别用三种查询�Q�每��?1,10,20 id�Q�，在不同�ƈ发下��试3�ơ，每次发出 1w �ơ查询。第一行数据�ؓ(f��)所有线�E�篏加时��_(d��)��单位 ms�Q�，�W�二行数据�ؓ(f��)吞吐�?1w /(total time / thread num))。测试中内存使用慢慢增加�Q�所以后面的数据可能比较高效的（高效的环境）�?br />
从上表看�Q?0 - 20�U�程比较高的吞吐量。看到内存��用，前提��是索引加蝲到内存中�Q��ƈ有些内存作�ؓ(f��)�~�存�?br />
下面有个索引查询优化�?pdf�?br />
Indexing and Query Optimizer

Indexing and Query Optimizer (Aaron Staple)
ps:

默认 mongodb 服务器只�?0个�ƈ发，如果要提高它的连接数�Q�可以用 --maxConns num 来提高它的接收�ƈ发的数据�?br />
mongodb �?java 驱动默认最多只�?10 �q�发�q�接池。要提高它，可以�?mongo.jar 的环境中加入 MONGO.POOLSIZE �pȝ��参数�Q�如 java -DMONGO.POOLSIZE=50 ...

paulwong 2015-01-06 23:39 发表评论

关于MongoDB最大连接数的查看与修改

paulwong — Tue, 06 Jan 2015 14:10:00 GMT

在Linux�q�_��下，无论�?4位或�?2位的MongoDB默认最大连接数都是819�Q�W(xu��)IN�q�_��不知道，估计也没有�h�?WIN�q�_��下��用MongoDB做生产环�?br />

[root@DELL113 mongodb-linux-i686-2.4.1]# mongo admin -u root -p password
MongoDB shell version: 2.4.1
connecting to: 192.168.6.42/admin
> db.serverStatus().connections
{ "current" : 1, "available" : 818, "totalCreated" : NumberLong(1) }

途中available昄��818��了一个，表示�I�闲的。current表示已经占用了的�q�接敎ͼ�两数一加就�{�于819�Q�如果我现在在连接一个，那么available��是817�Q�current��是2

[root@DELL113 mongodb-linux-i686-2.4.1]# ./bin/mongo 192.168.6.42
MongoDB shell version: 2.4.1
connecting to: 192.168.6.42/test
> db.serverStatus().connections
{ "current" : 1, "available" : 818, "totalCreated" : NumberLong(1) }
> db.serverStatus().connections
{ "current" : 2, "available" : 817, "totalCreated" : NumberLong(2) }

819个连接数对于一般的站点我认为已�l�够用，�q�且都是现连现取现断。但�q�个�q�接��C��可以修改�Q�只要在启动的时候加�?-maxConns卛_��
服务器启�?br />

[root@lee mongodb-linux-x86_64-2.4.1]# ./bin/mongod --dbpath=/root/db --maxConns=2000
Wed Apr 3 11:06:21.905 [initandlisten] MongoDB starting : pid=2812 port=27017 dbpath=/root/db 64-bit host=lee
Wed Apr 3 11:06:21.957 [initandlisten] db version v2.4.1
Wed Apr 3 11:06:21.957 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110
Wed Apr 3 11:06:21.957 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Apr 3 11:06:21.957 [initandlisten] allocator: tcmalloc
Wed Apr 3 11:06:21.957 [initandlisten] options: { dbpath: "/root/db", maxConns: 2000 }
Wed Apr 3 11:06:21.982 [initandlisten] journal dir=/root/db/journal
Wed Apr 3 11:06:21.982 [initandlisten] recover : no journal files present, no recovery needed
Wed Apr 3 11:06:22.297 [initandlisten] preallocateIsFaster=true 2.62
Wed Apr 3 11:06:22.717 [initandlisten] --maxConns too high, can only handle 819
Wed Apr 3 11:06:22.724 [initandlisten] waiting for connections on port 27017
Wed Apr 3 11:06:22.725 [websvr] admin web console waiting for connections on port 28017
Wed Apr 3 11:06:25.126 [initandlisten] connection accepted from 192.168.4.86:53917 #1 (1 connection now open)

查询最大连接数

发现�q�是819�Q�其实是Linux默认�q�程能打开最大文件数有关�Q�可以通过ulimit 解决

[root@lee mongodb-linux-x86_64-2.4.1]# ulimit -n 2500
[root@lee mongodb-linux-x86_64-2.4.1]# ./bin/mongod --dbpath=/root/db --maxConns=2000
Wed Apr 3 11:11:07.013 [initandlisten] MongoDB starting : pid=2930 port=27017 dbpath=/root/db 64-bit host=lee
Wed Apr 3 11:11:07.013 [initandlisten] db version v2.4.1
Wed Apr 3 11:11:07.013 [initandlisten] git version: 1560959e9ce11a693be8b4d0d160d633eee75110
Wed Apr 3 11:11:07.013 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Wed Apr 3 11:11:07.013 [initandlisten] allocator: tcmalloc
Wed Apr 3 11:11:07.013 [initandlisten] options: { dbpath: "/root/db", maxConns: 2000 }
Wed Apr 3 11:11:07.031 [initandlisten] journal dir=/root/db/journal
Wed Apr 3 11:11:07.031 [initandlisten] recover : no journal files present, no recovery needed
Wed Apr 3 11:11:07.170 [initandlisten] waiting for connections on port 27017
Wed Apr 3 11:11:07.171 [websvr] admin web console waiting for connections on port 28017
Wed Apr 3 11:11:10.076 [initandlisten] connection accepted from 192.168.4.86:53161 #1 (1 connection now open)

再查看最大连接数�Q�搞�?br />

关于ulimit的更多知识大家可以去�|�上��(g��)索检�?br />
客户端程序通常是通过DRIVER来链接，�׃��每次建立链接的成本都挺高�Q�因此都用链接池来实玎ͼ�SPRING DATA MONGODB中是如下配置

mongo.dbname=cms
#�U�程池的大小
mongo.connectionsPerHost=100
#�q�个*mongo.connectionsPerHost则是如果链接数大�?00的等待xttk�?/span>
mongo.threadsAllowedToBlockForConnectionMultiplier=4
#�{�待�U�程的等待时�?/span>
mongo.maxWaitTime=1500
mongo.socketTimeout=1500
mongo.connectTimeout=1000
mongo.autoConnectRetry=true
mongo.socketKeepAlive=true
mongo.slaveOk=true

autoConnectRetry simply means the driver will automatically attempt to reconnect to the server(s) after unexpected disconnects. In production environments you usually want this set to true.
connectionsPerHost are the amount of physical connections a single Mongo instance (it's singleton so you usually have one per application) can establish to a mongod/mongos process. At time of writing the java driver will establish this amount of connections eventually even if the actual query throughput is low (in order words you will see the "conn" statistic in mongostat rise until it hits this number per app server).
There is no need to set this higher than 100 in most cases but this setting is one of those "test it and see" things. Do note that you will have to make sure you set this low enough so that the total amount of connections to your server do not exceed
db.serverStatus().connections.available
In production we currently have this at 40.
connectTimeout. As the name suggest number of milliseconds the driver will wait before a connection attempt is aborted. Set timeout to something long (15-30 seconds) unless there's a realistic, expected chance this will be in the way of otherwise succesful connection attempts. Normally if a connection attempt takes longer than a couple of seconds your network infrastructure isn't capable of high throughput.
maxWaitTime. Number of ms a thread will wait for a connection to become available on the connection pool, and raises an exception if this does not happen in time. Keep default.
socketTimeout. Standard socket timeout value. Set to 60 seconds (60000).
threadsAllowedToBlockForConnectionMultiplier. Multiplier for connectionsPerHost that denotes the number of threads that are allowed to wait for connections to become available if the pool is currently exhausted. This is the setting that will cause the "com.mongodb.DBPortPool$SemaphoresOut: Out of semaphores to get db connection" exception. It will throw this exception once this thread queue exceeds the threadsAllowedToBlockForConnectionMultiplier value. For example, if the connectionsPerHost is 10 and this value is 5 up to 50 threads can block before the aforementioned exception is thrown.
If you expect big peaks in throughput that could cause large queues temporarily increase this value. We have it at 1500 at the moment for exactly that reason. If your query load consistently outpaces the server you should just improve your hardware/scaling situation accordingly.
readPreference. (UPDATED, 2.8+) Used to determine the default read preference and replaces "slaveOk". Set up a ReadPreference through one of the class factory method. A full description of the most common settings can be found at the end of this post
w. (UPDATED, 2.6+) This value determines the "safety" of the write. When this value is -1 the write will not report any errors regardless of network or database errors. WriteConcern.NONE is the appropriate predefined WriteConcern for this. If w is 0 then network errors will make the write fail but mongo errors will not. This is typically referred to as "fire and forget" writes and should be used when performance is more important than consistency and durability. Use WriteConcern.NORMAL for this mode.
If you set w to 1 or higher the write is considered safe. Safe writes perform the write and follow it up by a request to the server to make sure the write succeeded or retrieve an error value if it did not (in other words, it sends a getLastError() command after you write). Note that until this getLastError() command is completed the connection is reserved. As a result of that and the additional command the throughput will be signficantly lower than writes with w <= 0. With a w value of exactly 1 MongoDB guarantees the write succeeded (or verifiably failed) on the instance you sent the write to.
In the case of replica sets you can use higher values for w whcih tell MongoDB to send the write to at least "w" members of the replica set before returning (or more accurately, wait for the replication of your write to "w" members). You can also set w to the string "majority" which tells MongoDB to perform the write to the majority of replica set members (WriteConcern.MAJORITY). Typicall you should set this to 1 unless you need raw performance (-1 or 0) or replicated writes (>1). Values higher than 1 have a considerable impact on write throughput.
fsync. Durability option that forces mongo to flush to disk after each write when enabled. I've never had any durability issues related to a write backlog so we have this on false (the default) in production.
j *(NEW 2.7+)*. Boolean that when set to true forces MongoDB to wait for a successful journaling group commit before returning. If you have journaling enabled you can enable this for additional durability. Refer to http://www.mongodb.org/display/DOCS/Journaling to see what journaling gets you (and thus why you might want to enable this flag).

ReadPreference The ReadPreference class allows you to configure to what mongod instances queries are routed if you are working with replica sets. The following options are available :

ReadPreference.primary() : All reads go to the repset primary member only. Use this if you require all queries to return consistent (the most recently written) data. This is the default.
ReadPreference.primaryPreferred() : All reads go to the repset primary member if possible but may query secondary members if the primary node is not available. As such if the primary becomes unavailable reads become eventually consistent, but only if the primary is unavailable.
ReadPreference.secondary() : All reads go to secondary repset members and the primary member is used for writes only. Use this only if you can live with eventually consistent reads. Additional repset members can be used to scale up read performance although there are limits to the amount of (voting) members a repset can have.
ReadPreference.secondaryPreferred() : All reads go to secondary repset members if any of them are available. The primary member is used exclusively for writes unless all secondary members become unavailable. Other than the fallback to the primary member for reads this is the same as ReadPreference.secondary().
ReadPreference.nearest() : Reads go to the nearest repset member available to the database client. Use only if eventually consistent reads are acceptable. The nearest member is the member with the lowest latency between the client and the various repset members. Since busy members will eventually have higher latencies this should also automatically balance read load although in my experience secondary(Preferred) seems to do so better if member latencies are relatively consistent.

Note : All of the above have tag enabled versions of the same method which return TaggableReadPreference instances instead. A full description of replica set tags can be found here :Replica Set Tags

参考网址�Q?br />http://api.mongodb.org/java/2.10.1/com/mongodb/MongoClientOptions.Builder.html#connectionsPerHost(int)
https://github.com/spring-projects/spring-data-mongodb/blob/master/spring-data-mongodb/src/main/resources/org/springframework/data/mongodb/config/spring-mongo-1.5.xsd

paulwong 2015-01-06 22:10 发表评论

Architecture for Redis cache & Mongo for persistence

paulwong — Sun, 04 Jan 2015 07:50:00 GMT

Caching with Spring Data Redis
http://www.javacodegeeks.com/2013/02/caching-with-spring-data-redis.html

Architecture for Redis cache & Mongo for persistence
http://stackoverflow.com/questions/11218941/architecture-for-redis-cache-mongo-for-persistence

MongoDB with redis
http://stackoverflow.com/questions/10696463/mongodb-with-redis/10721249#10721249

Caching Data in Spring Using Redis
http://caseyscarborough.com/blog/2014/12/18/caching-data-in-spring-using-redis/

Springside Redis
https://github.com/springside/springside4/wiki/Redis

Spring Cache注解+Redis
http://hanqunfeng.iteye.com/blog/2176172

paulwong 2015-01-04 15:50 发表评论