[关闭]
@liyuj 2018-01-27T10:02:54.000000Z 字数 42521 阅读 3311

Apache-Ignite-2.3.0-中文开发手册

6.Cassandra集成

6.1.Ignite和Apache Cassandra

6.1.1.摘要

对于过期的缓存记录,通过使用Cassandra作为持久化存储,Ignite的Cassandra模块为缓存实现了一个CacheStore。
他在功能上和CacheJdbcBlobStore以及CacheJdbcPojoStore的方式几乎是相同的,但是又提供了如下的好处;

  1. 使用Apache Cassandra,它是一个高性能和高可扩展的键值存储;

    SQL查询
    注意,为了执行SQL查询,如果使用了外部的数据库,需要在Ignite缓存中持有所有的数据。
    或者也可以使用Ignite原生的持久化-这是一个分布式的、支持ACID以及兼容SQL的磁盘存储,它可以在存储于内存和磁盘上的数据执行SQL查询。

  2. 对于CacheStore的批量操作loadAll(),writeAll(),deleteAll(),使用Cassandra的异步查询 ,可以提供非常高的性能;

  3. 如果Cassandra中不存在会自动创建所有必要的表(以及键空间),也会为将存储为POJOs的Ignite键值自动检测所有必要的字段,并且创建相应的表结构。因此无需关注Cassandra的表创建DDL语法以及Java到Cassandra的类型映射细节。也可以使用@QuerySqlField注解为Cassandra表列提供配置(列名、索引、排序);
  4. 可以有选择地为将创建的Cassandra表和键空间指定配置(复制因子、复制策略、Bloom过滤器等);
  5. 组合BLOB和POJO存储的功能,可以根据喜好存储从Ignite缓存来的键值对(作为BLOB或者POJO);
  6. 对于键值支持JavaKryo序列化,它会以BLOB形式存储于Cassandra;
  7. 通过为特定的Ignite缓存指定持久化配置,或者通过使用@QuerySqlField(index = true)注解自动进行配置的检测,支持Cassandra的第二索引(包括定制索引);
  8. 通过持久化配置,或者通过使用@QuerySqlField(descending = true)注解自动进行配置的检测,支持Cassandra集群键字段的排序;
  9. 对于POJO的键类,如果它的属性之一加注了@AffinityKeyMapped注解,也会支持关系的并置,这时,Ignite缓存中存储在某个节点上的键值对,也会存储(并置)于Cassandra中的同一个节点上。

6.2.基本概念

要将Cassandra设置为一个持久化存储,需要将Ignite缓存的CacheStoreFactory设置为org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory
可以像下面这样通过Spring进行配置:

  1. <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
  2. <property name="cacheConfiguration">
  3. <list>
  4. ...
  5. <!-- Configuring persistence for "cache1" cache -->
  6. <bean class="org.apache.ignite.configuration.CacheConfiguration">
  7. <property name="name" value="cache1"/>
  8. <!-- Tune on Read-Through and Write-Through mode -->
  9. <property name="readThrough" value="true"/>
  10. <property name="writeThrough" value="true"/>
  11. <!-- Specifying CacheStoreFactory -->
  12. <property name="cacheStoreFactory">
  13. <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
  14. <!-- Datasource configuration bean which is responsible for Cassandra connection details -->
  15. <property name="dataSourceBean" value="cassandraDataSource"/>
  16. <!-- Persistent settings bean which is responsible for the details of how objects will be persisted to Cassandra -->
  17. <property name="persistenceSettingsBean" value="cache1_persistence_settings"/>
  18. </bean>
  19. </property>
  20. </bean>
  21. ...
  22. </list>
  23. ...
  24. </property>
  25. </bean>

这里有两个主要的属性需要为CassandraCacheStoreFactory设置:

下面的章节中这两个Bean及其配置会详细地描述。

6.2.1.DataSourceBean

这个bean存储了Cassandra数据库与连接和CRUD操作有关的所有必要信息,下面的表格中显示了所有的属性:

属性 默认值 描述
user 连接Cassandra的用户名
password 连接Cassandra的用户密码
credentials 提供userpassword的Credentials Bean
authProvider 接入Cassandra时使用指定的AuthProvider,当自定义身份认证体系准备就绪时,使用这个方法。
port 接入Cassandra时使用的端口(如果没有在连接点中提供)
contactPoints Cassandra连接时使用的连接点数组(hostaname:[port]
maxSchemaAgreementWaitSeconds 10秒 DDL查询返回前架构协议的最大等待时间
protocolVersion 3 指定使用Cassandra驱动协议的哪个版本(有助于旧版本Cassandra的后向兼容)。
compression 传输中使用的压缩格式,支持的压缩格式包括:snappylz4
useSSL false 是否启用SSL
sslOptions false 是否使用提供的选项启用SSL
collectMetrix false 是否启用指标收集
jmxReporting false 是否启用JMX的指标报告
fetchSize 指定查询获取大小,获取大小控制一次获取的结果集的数量
readConsistency 指定READ查询的一致性级别
writeConsistency 指定WRITE/DELETE/UPDATE查询的一致性级别
loadBalancingPolicy TokenAwarePolicy 指定要使用的负载平衡策略
reconnectionPolicy ExponentialReconnectionPolicy 指定要使用的重连策略
retryPolicy DefaultRetryPolicy 指定要使用的重试策略
addressTranslater IdentityTranslater 指定要使用的地址转换器
speculativeExecutionPolicy NoSpeculativeExecutionPolicy 指定要使用 的推理执行策略
poolingOptions 指定连接池选项
socketOptions 指定保持到Cassandra主机的连接的底层Socket选项
nettyOptions 允许客户端定制Cassandra驱动底层Netty层的钩子

6.2.2.PersistenceSettingsBean

这个bean存储了对象(键和值)如何持久化到Cassandra数据库的所有细节信息(键空间、表、分区选项、POJO字段映射等)。
org.apache.ignite.cache.store.cassandra.utils.persistence.KeyValuePersistenceSettings的构造器可以通过如下方式创建这个Bean,从一个包含特定结构的XML配置文档的字符串(看下面的代码),或者指向XML文档的资源。
下面是一个XML配置文档的常规示例(持久化描述符),它指定了Ignite缓存的键和值如何序列化/反序列化到/从Cassandra:

  1. <!--
  2. Root container for persistence settings configuration.
  3. Note: required element
  4. Attributes:
  5. 1) keyspace [required] - specifies keyspace for Cassandra tables which should be used to store key/value pairs
  6. 2) table [required] - specifies Cassandra table which should be used to store key/value pairs
  7. 3) ttl [optional] - specifies expiration period for the table rows (in seconds)
  8. -->
  9. <persistence keyspace="my_keyspace" table="my_table" ttl="86400">
  10. <!--
  11. Specifies Cassandra keyspace options which should be used to create provided keyspace if it doesn't exist.
  12. Note: optional element
  13. -->
  14. <keyspaceOptions>
  15. REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3}
  16. AND DURABLE_WRITES = true
  17. </keyspaceOptions>
  18. <!--
  19. Specifies Cassandra table options which should be used to create provided table if it doesn't exist.
  20. Note: optional element
  21. -->
  22. <tableOptions>
  23. comment = 'A most excellent and useful table'
  24. AND read_repair_chance = 0.2
  25. </tableOptions>
  26. <!--
  27. Specifies persistent settings for Ignite cache keys.
  28. Note: required element
  29. Attributes:
  30. 1) class [required] - java class name for Ignite cache key
  31. 2) strategy [required] - one of three possible persistent strategies:
  32. a) PRIMITIVE - stores key value as is, by mapping it to Cassandra table column with corresponding type.
  33. Should be used only for simple java types (int, long, String, double, Date) which could be mapped
  34. to corresponding Cassadra types.
  35. b) BLOB - stores key value as BLOB, by mapping it to Cassandra table column with blob type.
  36. Could be used for any java object. Conversion of java object to BLOB is handled by "serializer"
  37. which could be specified in serializer attribute (see below).
  38. c) POJO - stores each field of an object as a column having corresponding type in Cassandra table.
  39. Provides ability to utilize Cassandra secondary indexes for object fields.
  40. 3) serializer [optional] - specifies serializer class for BLOB strategy. Shouldn't be used for PRIMITIVE and
  41. POJO strategies. Available implementations:
  42. a) org.apache.ignite.cache.store.cassandra.utils.serializer.JavaSerializer - uses standard Java
  43. serialization framework
  44. b) org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer - uses Kryo
  45. serialization framework
  46. 4) column [optional] - specifies column name for PRIMITIVE and BLOB strategies where to store key value.
  47. If not specified column having 'key' name will be used. Shouldn't be used for POJO strategy.
  48. -->
  49. <keyPersistence class="org.mycompany.MyKeyClass" strategy="..." serializer="..." column="...">
  50. <!--
  51. Specifies partition key fields if POJO strategy used.
  52. Note: optional element, only required for POJO strategy in case you want to manually specify
  53. POJO fields to Cassandra columns mapping, instead of relying on dynamic discovering of
  54. POJO fields and mapping them to the same columns of Cassandra table.
  55. -->
  56. <partitionKey>
  57. <!--
  58. Specifies mapping from POJO field to Cassandra table column.
  59. Note: required element
  60. Attributes:
  61. 1) name [required] - POJO field name
  62. 2) column [optional] - Cassandra table column name. If not specified lowercase
  63. POJO field name will be used.
  64. -->
  65. <field name="companyCode" column="company" />
  66. ...
  67. ...
  68. </partitionKey>
  69. <!--
  70. Specifies cluster key fields if POJO strategy used.
  71. Note: optional element, only required for POJO strategy in case you want to manually specify
  72. POJO fields to Cassandra columns mapping, instead of relying on dynamic discovering of
  73. POJO fields and mapping them to the same columns of Cassandra table.
  74. -->
  75. <clusterKey>
  76. <!--
  77. Specifies mapping from POJO field to Cassandra table column.
  78. Note: required element
  79. Attributes:
  80. 1) name [required] - POJO field name
  81. 2) column [optional] - Cassandra table column name. If not specified lowercase
  82. POJO field name will be used.
  83. 3) sort [optional] - specifies sort order (asc or desc)
  84. -->
  85. <field name="personNumber" column="number" sort="desc"/>
  86. ...
  87. ...
  88. </clusterKey>
  89. </keyPersistence>
  90. <!--
  91. Specifies persistent settings for Ignite cache values.
  92. Note: required element
  93. Attributes:
  94. 1) class [required] - java class name for Ignite cache value
  95. 2) strategy [required] - one of three possible persistent strategies:
  96. a) PRIMITIVE - stores key value as is, by mapping it to Cassandra table column with corresponding type.
  97. Should be used only for simple java types (int, long, String, double, Date) which could be mapped
  98. to corresponding Cassadra types.
  99. b) BLOB - stores key value as BLOB, by mapping it to Cassandra table column with blob type.
  100. Could be used for any java object. Conversion of java object to BLOB is handled by "serializer"
  101. which could be specified in serializer attribute (see below).
  102. c) POJO - stores each field of an object as a column having corresponding type in Cassandra table.
  103. Provides ability to utilize Cassandra secondary indexes for object fields.
  104. 3) serializer [optional] - specifies serializer class for BLOB strategy. Shouldn't be used for PRIMITIVE and
  105. POJO strategies. Available implementations:
  106. a) org.apache.ignite.cache.store.cassandra.utils.serializer.JavaSerializer - uses standard Java
  107. serialization framework
  108. b) org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer - uses Kryo
  109. serialization framework
  110. 4) column [optional] - specifies column name for PRIMITIVE and BLOB strategies where to store value.
  111. If not specified column having 'value' name will be used. Shouldn't be used for POJO strategy.
  112. -->
  113. <valuePersistence class="org.mycompany.MyValueClass" strategy="..." serializer="..." column="">
  114. <!--
  115. Specifies mapping from POJO field to Cassandra table column.
  116. Note: required element
  117. Attributes:
  118. 1) name [required] - POJO field name
  119. 2) column [optional] - Cassandra table column name. If not specified lowercase
  120. POJO field name will be used.
  121. 3) static [optional] - boolean flag which specifies that column is static withing a given partition
  122. 4) index [optional] - boolean flag specifying that secondary index should be created for the field
  123. 5) indexClass [optional] - custom index java class name if you want to use custom index
  124. 6) indexOptions [optional] - custom index options
  125. -->
  126. <field name="firstName" column="first_name" static="..." index="..." indexClass="..." indexOptions="..."/>
  127. ...
  128. ...
  129. </valuePersistence>
  130. </persistence>

下面会提供关于持久化描述符配置及其元素的所有细节信息。
persistence

必要元素
持久化配置的根容器。

属性 必需 描述
keyspace 存储键值对的Cassandra表的键空间,如果键空间不存在会创建它(如果指定的Cassandra账户持有正确的权限)。
table 存储键值对的Cassandra表,如果表不存在会创建它(如果指定的Cassandra账户持有正确的权限)。
ttl 表数据行的到期时间(秒),要了解有关Cassandra ttl的详细信息,可以参照到期数据

keyspaceOptions

可选元素
创建在持久化配置容器中配置的keyspace属性指定的Cassandra键空间时的可选项。

键空间只有在不存在时才会被创建,并且连接到Cassandra的账户要持有正确的权限。
这个XML元素指定的文本只是创建键空间的Cassandra DDL语句中在WITH关键字之后的一段代码。
tableOptions

可选元素
创建在持久化配置容器中配置的table属性指定的表时的可选项。

表只有在不存在时才会被创建,并且连接到Cassandra的账户要持有正确的权限。
这个XML元素指定的文本只是创建表的Cassandra DDL语句中在WITH关键字之后的一段代码。
keyPersistence

必要元素
Ignite缓存键的持久化配置。

这些属性指定了从Ignite缓存中对象如何存储/加载到/从Cassandra表。

属性 必需 描述
class Ignite缓存键的Java类名。
strategy 指定三个可能的持久化策略之一(看下面的描述),它会控制对象如何存储/加载到/从Cassandra表。
serializer BLOB策略的序列化器类(可用的实现看下面),PRIMITIVE和POJO策略时无法使用。
column PRIMITIVE和BLOB策略时存储键的列名,如果不指定,列名为key,对于POJO策略属性无需指定。

持久化策略

名称 描述
PRIMITIVE 存储对象,通过对应的类型将其映射到Cassandra表列中,只能使用简单的Java类型(int、long、String、double、Date),他们会直接映射到对应的Cassandra类型上,要了解详细的Java到Cassandra的类型映射,点击这里
BLOB 将对象存储为BLOB,使用BLOB类型将其映射到Cassandra表列,可以使用任何Java对象,Java对象到BLOB的转换是由keyPersistence容器中的serializer属性指定的序列化器处理的。
POJO 将对象的每个属性按照对应的类型存储到Cassandra的表中,对于对象的属性,提供了利用Cassandra第二索引的能力,只能用于遵守Java Bean规范的POJO对象,并且它的属性都是基本Java类型,它们会直接映射到对应的Cassandra类型上。

可用的序列化器实现

类名 描述
org.apache.ignite.cache.store.cassandra.utils.serializer.JavaSerializer 使用标准的Java序列化框架
org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer 使用Kryo序列化框架

如果使用了PRIMITIVEBLOB持久化策略,那么是不需要指定keyPersistence标签的内部元素的,这样的原因是,这两个策略中整个对象都被持久化到Cassandra表中的一列(可以通过column指定)。
如果使用POJO持久化策略,那么有两个策略:

下面两个章节会详细描述分区集群键字段映射的细节(如果选择了上面列表的第二个选项)。
partitionKey

可选元素
field元素的容器,用于指定Cassandra的分区键。

定义了Ignite缓存的键对象字段(在它里面),他会被用作Cassandra表的分区键,并且指定了到表列的字段映射。
映射是通过<field>标签设定的,它有如下的属性:

属性 必需 描述
name POJO对象字段名
column Cassandra表列名,如果不指定,会使用POJO字段名的小写形式

clusterKey

可选元素
field元素的容器,用于指定Cassandra的集群键。

定义了Ignite缓存的键对象字段(在它里面),他会被用作Cassandra表的集群键,并且指定了到表列的字段映射。
映射是通过<field>标签设定的,它有如下的属性:

属性 必需 描述
name POJO对象字段名
column Cassandra表列名,如果不指定,会使用POJO字段名的小写形式
sort 指定字段排序规则(升序或者降序

valuePersistence

必要元素
Ignite缓存值的持久化配置。

这些设置指定了Ignite缓存的值对象如何存储/加载到/从Cassandra表。
这些设置的属性看上去和对应的Ignite缓存键的设定很像。

属性 必需 描述
class Ignite缓存值的Java类名。
strategy 指定三个可能的持久化策略之一(看下面的描述),它会控制对象如何存储/加载到/从Cassandra表。
serializer BLOB策略的序列化器类(可用的实现看下面),PRIMITIVE和POJO策略时无法使用。
column PRIMITIVE和BLOB策略时存储值的列名,如果不指定,列名为value,对于POJO策略属性无需指定。

持久化策略(与键的持久化策略一致):

名称 描述
PRIMITIVE 存储对象,通过对应的类型将其映射到Cassandra表列中,只能使用简单的Java类型(int、long、String、double、Date),他们会直接映射到对应的Cassandra类型上,要了解详细的Java到Cassandra的类型映射,点击这里
BLOB 将对象存储为BLOB,使用BLOB类型将其映射到Cassandra表列,可以使用任何Java对象,Java对象到BLOB的转换是由valuePersistence容器中的serializer属性指定的序列化器处理的。
POJO 将对象的每个属性按照对应的类型存储到Cassandra的表中,对于对象的属性,提供了利用Cassandra第二索引的能力,只能用于遵守Java Bean规范的POJO对象,并且它的属性都是基本Java类型,它们会直接映射到对应的Cassandra类型上。

可用的序列化器实现

类名 描述
org.apache.ignite.cache.store.cassandra.utils.serializer.JavaSerializer 使用标准的Java序列化框架
org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer 使用Kryo序列化框架

如果使用了PRIMITIVEBLOB持久化策略,那么是不需要指定valuePersistence标签的内部元素的,这样的原因是,这两个策略中整个对象都被持久化到Cassandra表中的一列(可以通过column指定)。
如果使用POJO持久化策略,那么有两个策略(与键的选项相同):

如果选择了上述的第二个选项,那么需要使用<field>标签指定POJO字段到Cassandra表列的映射,这个标签有如下的属性:

属性 必需 描述
name POJO对象字段名
column Cassandra表列名,如果不指定,会使用POJO字段名的小写形式
static 布尔类型标志,它指定了在一个分区内列是否为静态的
index 布尔类型标志,指定了对于特定字段是否要创建第二索引
indexClass 自定义索引Java类名,如果要使用自定义索引的话
indexOptions 自定义索引选项

6.3.示例

就像上一章描述的那样,要将Cassandra配置为缓存存储,需要将Ignite缓存的CacheStoreFactory设置为org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory
下面是一个Ignite将Cassandra配置为缓存存储的典型配置示例,即使他看上去很复杂也不用担心,我们会一步一步深入每一个配置项,这个示例来自于Cassandra模块源代码的单元测试资源文件test/resources/org/apache/ignite/tests/persistence/blob/ignite-config.xml
XML:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <beans xmlns="http://www.springframework.org/schema/beans"
  3. xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4. xsi:schemaLocation="
  5. http://www.springframework.org/schema/beans
  6. http://www.springframework.org/schema/beans/spring-beans.xsd">
  7. <!-- Cassandra connection settings -->
  8. <import resource="classpath:org/apache/ignite/tests/cassandra/connection-settings.xml" />
  9. <!-- Persistence settings for 'cache1' -->
  10. <bean id="cache1_persistence_settings" class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
  11. <constructor-arg type="org.springframework.core.io.Resource" value="classpath:org/apache/ignite/tests/persistence/blob/persistence-settings-1.xml" />
  12. </bean>
  13. <!-- Persistence settings for 'cache2' -->
  14. <bean id="cache2_persistence_settings" class="org.apache.ignite.cache.store.cassandra.persistence.KeyValuePersistenceSettings">
  15. <constructor-arg type="org.springframework.core.io.Resource" value="classpath:org/apache/ignite/tests/persistence/blob/persistence-settings-3.xml" />
  16. </bean>
  17. <!-- Ignite configuration -->
  18. <bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
  19. <property name="cacheConfiguration">
  20. <list>
  21. <!-- Configuring persistence for "cache1" cache -->
  22. <bean class="org.apache.ignite.configuration.CacheConfiguration">
  23. <property name="name" value="cache1"/>
  24. <property name="readThrough" value="true"/>
  25. <property name="writeThrough" value="true"/>
  26. <property name="cacheStoreFactory">
  27. <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
  28. <property name="dataSourceBean" value="cassandraAdminDataSource"/>
  29. <property name="persistenceSettingsBean" value="cache1_persistence_settings"/>
  30. </bean>
  31. </property>
  32. </bean>
  33. <!-- Configuring persistence for "cache2" cache -->
  34. <bean class="org.apache.ignite.configuration.CacheConfiguration">
  35. <property name="name" value="cache2"/>
  36. <property name="readThrough" value="true"/>
  37. <property name="writeThrough" value="true"/>
  38. <property name="cacheStoreFactory">
  39. <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
  40. <property name="dataSourceBean" value="cassandraAdminDataSource"/>
  41. <property name="persistenceSettingsBean" value="cache2_persistence_settings"/>
  42. </bean>
  43. </property>
  44. </bean>
  45. </list>
  46. </property>
  47. <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
  48. <property name="discoverySpi">
  49. <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
  50. <property name="ipFinder">
  51. <!--
  52. Ignite provides several options for automatic discovery that can be used
  53. instead os static IP based discovery. For information on all options refer
  54. to our documentation: http://apacheignite.readme.io/docs/cluster-config
  55. -->
  56. <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
  57. <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
  58. <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
  59. <property name="addresses">
  60. <list>
  61. <!-- In distributed environment, replace with actual host IP address. -->
  62. <value>127.0.0.1:47500..47509</value>
  63. </list>
  64. </property>
  65. </bean>
  66. </property>
  67. </bean>
  68. </property>
  69. </bean>
  70. </beans>

在这个示例中,配置了两个Ignite缓存:cache1cache2,下面会看配置的细节。
这两个缓存非常接近(cache1cache2),看起来像这样:
XML:

  1. <bean class="org.apache.ignite.configuration.CacheConfiguration">
  2. <property name="name" value="cache1"/>
  3. <property name="readThrough" value="true"/>
  4. <property name="writeThrough" value="true"/>
  5. <property name="cacheStoreFactory">
  6. <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
  7. <property name="dataSourceBean" value="cassandraAdminDataSource"/>
  8. <property name="persistenceSettingsBean" value="cache1_persistence_settings"/>
  9. </bean>
  10. </property>
  11. </bean>

首先,我们看到通读和通写选项已经启用了:
XML:

  1. <property name="readThrough" value="true"/>
  2. <property name="writeThrough" value="true"/>

如果希望为过期条目使用持久化存储,这个对于Ignite缓存就是必要的。
如果希望异步更新持久化存储的话,也可以有选择地配置后写参数。
XML:

  1. <property name="writeBehindEnabled" value="true"/>

下一个重要的事就是CacheStoreFactory的配置:
XML:

  1. <property name="cacheStoreFactory">
  2. <bean class="org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory">
  3. <property name="dataSourceBean" value="cassandraAdminDataSource"/>
  4. <property name="persistenceSettingsBean" value="cache1_persistence_settings"/>
  5. </bean>
  6. </property>

可以看到将org.apache.ignite.cache.store.cassandra.CassandraCacheStoreFactory作为一个CacheStoreFactory,这使得Ignite缓存可以使用Cassandra作为持久化存储。对于CassandraCacheStoreFactory,需要指定两个必要的属性:

在这个示例中,cassandraAdminDataSource是一个datasource bean,可以使用如下的指令导入Ignite的缓存配置文件:

  1. <import resource="classpath:org/apache/ignite/tests/cassandra/connection-settings.xml" />

cache1_persistence_settings是一个持久化配置bean,他是在Ignite缓存配置文件中使用如下的方式配置的:
XML:

  1. <bean id="cache1_persistence_settings" class="org.apache.ignite.cache.store.cassandra.utils.persistence.KeyValuePersistenceSettings">
  2. <constructor-arg type="org.springframework.core.io.Resource" value="classpath:org/apache/ignite/tests/persistence/blob/persistence-settings-1.xml" />
  3. </bean>

现在我们可以从org/apache/ignite/tests/cassandra/connection-settings.xml测试资源文件中看一下cassandraAdminDataSource的设置:

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <beans xmlns="http://www.springframework.org/schema/beans"
  3. xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  4. xsi:schemaLocation="
  5. http://www.springframework.org/schema/beans
  6. http://www.springframework.org/schema/beans/spring-beans.xsd">
  7. <bean id="cassandraAdminCredentials" class="org.apache.ignite.tests.utils.CassandraAdminCredentials"/>
  8. <bean id="loadBalancingPolicy" class="com.datastax.driver.core.policies.RoundRobinPolicy"/>
  9. <bean id="contactPoints" class="org.apache.ignite.tests.utils.CassandraHelper" factory-method="getContactPointsArray"/>
  10. <bean id="cassandraAdminDataSource" class="org.apache.ignite.cache.store.cassandra.utils.datasource.DataSource">
  11. <property name="credentials" ref="cassandraAdminCredentials"/>
  12. <property name="contactPoints" ref="contactPoints"/>
  13. <property name="readConsistency" value="ONE"/>
  14. <property name="writeConsistency" value="ONE"/>
  15. <property name="loadBalancingPolicy" ref="loadBalancingPolicy"/>
  16. </bean>
  17. </beans>

最后,还没有描述的最后一个片段就是持久化设置的配置,我们可以从org/apache/ignite/tests/persistence/blob/persistence-settings-1.xml测试资源文件中看一下cache1_persistence_settings:
XML:

  1. <persistence keyspace="test1" table="blob_test1">
  2. <keyPersistence class="java.lang.Integer" strategy="PRIMITIVE" />
  3. <valuePersistence strategy="BLOB"/>
  4. </persistence>

在这个配置中,我们可以看到Cassandra的test1.blob_test1表会用于cache1缓存的键/值存储,缓存的键对象会以integer的形式存储于key列中,缓存的值对象会以blob的形式存储于value列中。
下一章会为不同类型的持久化策略提供持久化设置的示例。

6.3.1.示例1

Ignite缓存的持久化配置中,Integer类型的键在Cassandra中会以int的形式存储,String类型的值在Cassandra中会以text的形式存储。
XML:

  1. <persistence keyspace="test1" table="my_table">
  2. <keyPersistence class="java.lang.Integer" strategy="PRIMITIVE" column="my_key"/>
  3. <valuePersistence class="java.lang.String" strategy="PRIMITIVE" />
  4. </persistence>

键会存储于my_key列,值会存储于value列(如果column属性不指定的话会使用默认值)。

6.3.2.示例2

Ignite缓存的持久化配置中,Integer类型的键在Cassandra中会以int的形式存储,any类型的值(BLOB持久化策略中无需指定类型)在Cassandra中会以BLOB的形式存储,这个场景的唯一解决方案就是在Cassandra中将值存储为BLOB
XML:

  1. <persistence keyspace="test1" table="my_table">
  2. <keyPersistence class="java.lang.Integer" strategy="PRIMITIVE" />
  3. <valuePersistence strategy="BLOB"/>
  4. </persistence>

键会存储于key列(如果column属性不指定的话会使用默认值),值会存储于value列。

6.3.3.示例3

Ignite缓存的持久化配置中,Integer类型的键和any类型的值在Cassandra中都以BLOB的形式存储。

  1. <persistence keyspace="test1" table="my_table">
  2. <!-- By default Java standard serialization is going to be used -->
  3. <keyPersistence class="java.lang.Integer"
  4. strategy="BLOB"/>
  5. <!-- Kryo serialization specified to be used -->
  6. <valuePersistence class="org.apache.ignite.tests.pojos.Person"
  7. strategy="BLOB"
  8. serializer="org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer"/>
  9. </persistence>

键会存储于BLOB类型的key列,使用Java标准序列化,值会存储于BLOB类型的value列,使用Kryo序列化

6.3.4.示例4

Ignite缓存的持久化配置中,Integer类型的键在Cassandra中会以int的形式存储,自定义POJOorg.apache.ignite.tests.pojos.Person类型的值在动态分析后会被持久化到一组表列中,这样的话每个POJO字段都会被映射到相对应的表列,关于更多动态POJO字段发现的信息,可以查看上一章的介绍。
XML:

  1. <persistence keyspace="test1" table="my_table">
  2. <keyPersistence class="java.lang.Integer" strategy="PRIMITIVE"/>
  3. <valuePersistence class="org.apache.ignite.tests.pojos.Person" strategy="POJO"/>
  4. </persistence>

键会存储于int类型的key列。
我们可以假设org.apache.ignite.tests.pojos.Person类的实现如下:

  1. public class Person {
  2. private String firstName;
  3. private String lastName;
  4. private int age;
  5. private boolean married;
  6. private long height;
  7. private float weight;
  8. private Date birthDate;
  9. private List<String> phones;
  10. public void setFirstName(String name) {
  11. firstName = name;
  12. }
  13. public String getFirstName() {
  14. return firstName;
  15. }
  16. public void setLastName(String name) {
  17. lastName = name;
  18. }
  19. public String getLastName() {
  20. return lastName;
  21. }
  22. public void setAge(int age) {
  23. this.age = age;
  24. }
  25. public int getAge() {
  26. return age;
  27. }
  28. public void setMarried(boolean married) {
  29. this.married = married;
  30. }
  31. public boolean getMarried() {
  32. return married;
  33. }
  34. public void setHeight(long height) {
  35. this.height = height;
  36. }
  37. public long getHeight() {
  38. return height;
  39. }
  40. public void setWeight(float weight) {
  41. this.weight = weight;
  42. }
  43. public float getWeight() {
  44. return weight;
  45. }
  46. public void setBirthDate(Date date) {
  47. birthDate = date;
  48. }
  49. public Date getBirthDate() {
  50. return birthDate;
  51. }
  52. public void setPhones(List<String> phones) {
  53. this.phones = phones;
  54. }
  55. public List<String> getPhones() {
  56. return phones;
  57. }
  58. }

这时,Ignite缓存中org.apache.ignite.tests.pojos.Person类型的值会使用如下的动态配置映射规则持久化到一组Cassandra表列中:

POJO字段 表列 列类型
firstName firstname text
lastName lastname text
age age int
married married boolean
height height bigint
weight weight float
birthDate birthdate timestamp

从上表可以看出,phones字段不会被持久化到表中,这是应为他不是一个可以映射到对应Cassandra类型的简单Java类型,这种类型的字段只有在给这个对象类型手工指定所有的映射细节以及字段类型本身实现了java.io.Serializable接口时才会被持久化于Cassandra中。这时,字段会被持久化到一个单独的BLOB类型的表列。下个示例会看到更多的细节。

这个示例显示了,使用非常简单的配置,依托动态对象字段映射,就可以轻易地为POJO对象配置持久化。

6.3.5.示例5

Ignite缓存的持久化配置中,键是自定义的POJOorg.apache.ignite.tests.pojos.PersonId类型,值是自定义POJOorg.apache.ignite.tests.pojos.Person类型,基于手工指定的映射规则,都会被持久化到一组表列。
XML:

  1. <persistence keyspace="test1" table="my_table" ttl="86400">
  2. <!-- Cassandra keyspace options which should be used to create provided keyspace if it doesn't exist -->
  3. <keyspaceOptions>
  4. REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3}
  5. AND DURABLE_WRITES = true
  6. </keyspaceOptions>
  7. <!-- Cassandra table options which should be used to create provided table if it doesn't exist -->
  8. <tableOptions>
  9. comment = 'A most excellent and useful table'
  10. AND read_repair_chance = 0.2
  11. </tableOptions>
  12. <!-- Persistent settings for Ignite cache keys -->
  13. <keyPersistence class="org.apache.ignite.tests.pojos.PersonId" strategy="POJO">
  14. <!-- Partition key fields if POJO strategy used -->
  15. <partitionKey>
  16. <!-- Mapping from POJO field to Cassandra table column -->
  17. <field name="companyCode" column="company" />
  18. <field name="departmentCode" column="department" />
  19. </partitionKey>
  20. <!-- Cluster key fields if POJO strategy used -->
  21. <clusterKey>
  22. <!-- Mapping from POJO field to Cassandra table column -->
  23. <field name="personNumber" column="number" sort="desc"/>
  24. </clusterKey>
  25. </keyPersistence>
  26. <!-- Persistent settings for Ignite cache values -->
  27. <valuePersistence class="org.apache.ignite.tests.pojos.Person"
  28. strategy="POJO"
  29. serializer="org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer">
  30. <!-- Mapping from POJO field to Cassandra table column -->
  31. <field name="firstName" column="first_name" />
  32. <field name="lastName" column="last_name" />
  33. <field name="age" />
  34. <field name="married" index="true"/>
  35. <field name="height" />
  36. <field name="weight" />
  37. <field name="birthDate" column="birth_date" />
  38. <field name="phones" />
  39. </valuePersistence>
  40. </persistence>

这些配置看上去非常复杂,我们可以一步一步地分析它。
首先看一下根标签:
XML:

  1. <persistence keyspace="test1" table="my_table" ttl="86400">

它指定了Ignite缓存的键和值应该存储于test1.my_table表,并且每一条数据会在86400秒(24小时)后过期
然后可以看到关于Cassandra键空间的高级配置,在不存在时,这个配置会用于创建键空间。
XML:

  1. <keyspaceOptions>
  2. REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3}
  3. AND DURABLE_WRITES = true
  4. </keyspaceOptions>

然后通过对键空间配置的分析,可以看到只会用于表创建的高级配置。
XML:

  1. <tableOptions>
  2. comment = 'A most excellent and useful table'
  3. AND read_repair_chance = 0.2
  4. </tableOptions>

下一个章节说明了Ignite缓存的键如何持久化:
XML:

  1. <keyPersistence class="org.apache.ignite.tests.pojos.PersonId" strategy="POJO">
  2. <!-- Partition key fields if POJO strategy used -->
  3. <partitionKey>
  4. <!-- Mapping from POJO field to Cassandra table column -->
  5. <field name="companyCode" column="company" />
  6. <field name="departmentCode" column="department" />
  7. </partitionKey>
  8. <!-- Cluster key fields if POJO strategy used -->
  9. <clusterKey>
  10. <!-- Mapping from POJO field to Cassandra table column -->
  11. <field name="personNumber" column="number" sort="desc"/>
  12. </clusterKey>
  13. </keyPersistence>

我们假定org.apache.ignite.tests.pojos.PersonId的实现如下:

  1. public class PersonId {
  2. private String companyCode;
  3. private String departmentCode;
  4. private int personNumber;
  5. public void setCompanyCode(String code) {
  6. companyCode = code;
  7. }
  8. public String getCompanyCode() {
  9. return companyCode;
  10. }
  11. public void setDepartmentCode(String code) {
  12. departmentCode = code;
  13. }
  14. public String getDepartmentCode() {
  15. return departmentCode;
  16. }
  17. public void setPersonNumber(int number) {
  18. personNumber = number;
  19. }
  20. public int getPersonNumber() {
  21. return personNumber;
  22. }
  23. }

这时Ignite缓存中org.apache.ignite.tests.pojos.PersonId类型的键会使用如下的映射规则持久化到一组表示分区集群键的Cassandra表列:

POJO字段 表列 列类型
companyCode company text
departmentCode department text
personNumber number int

另外,(company, department)的列组合会用作为Cassandra的PARTITION键,number列会用作为倒序排列的集群键。
最后到最后一章,它指定了Ignite缓存值的持久化配置:
XML:

  1. <valuePersistence class="org.apache.ignite.tests.pojos.Person"
  2. strategy="POJO"
  3. serializer="org.apache.ignite.cache.store.cassandra.utils.serializer.KryoSerializer">
  4. <!-- Mapping from POJO field to Cassandra table column -->
  5. <field name="firstName" column="first_name" />
  6. <field name="lastName" column="last_name" />
  7. <field name="age" />
  8. <field name="married" index="true"/>
  9. <field name="height" />
  10. <field name="weight" />
  11. <field name="birthDate" column="birth_date" />
  12. <field name="phones" />
  13. </valuePersistence>

假定org.apache.ignite.tests.pojos.Person类和示例4的实现一样,这时Ignite缓存的org.apache.ignite.tests.pojos.Person类型的值会以如下的映射规则持久化到一组Cassandra表列:

POJO字段 表列 列类型
firstName first_name text
lastName last_name text
age age int
married married boolean
height height bigint
weight weight float
birthDate birth_date timestamp
phones phones blob

和示例4相比,我们可以看到,使用Kryo序列化器,phones字段会被序列化到blob类型的phones列。另外,Cassandra会为married列创建第二索引。

6.4.DDL生成器

Ignite Cassandra模块的一个好处是,无需关注Cassandra的表创建DDL语法以及Java到Cassandra的类型映射细节。
只需要创建指定了Ignite缓存的键和值如何序列化/反序列化到/从Cassandra的XML配置文档即可,基于这个设置,剩余的Cassandra键空间和表都会被自动创建,要让这一切运转起来,只需要:

在Cassandra的连接设置中,指定的用户要有足够的权限来创建键空间和表。

然而,因为严格的安全策略,某些环境中这是不可能的。这个场景的唯一解决方案就是向运维团队提供DDL脚本来创建所有必要的Cassandra键空间和表。
这就是使用DDL生成工具的确切场景,它会从一个持久化配置中生成DDL。
语法样例:

  1. java org.apache.ignite.cache.store.cassandra.utils.DDLGenerator /opt/dev/ignite/persistence-settings-1.xml /opt/dev/ignite/persistence-settings-2.xml

输出样例:

  1. -------------------------------------------------------------
  2. DDL for keyspace/table from file: /opt/dev/ignite/persistence-settings-1.xml
  3. -------------------------------------------------------------
  4. create keyspace if not exists test1
  5. with replication = {'class' : 'SimpleStrategy', 'replication_factor' : 3} and durable_writes = true;
  6. create table if not exists test1.primitive_test1
  7. (
  8. key int,
  9. value int,
  10. primary key ((key))
  11. );
  12. -------------------------------------------------------------
  13. DDL for keyspace/table from file: /opt/dev/ignite/persistence-settings-2.xml
  14. -------------------------------------------------------------
  15. create keyspace if not exists test1
  16. with REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 3} AND DURABLE_WRITES = true;
  17. create table if not exists test1.pojo_test3
  18. (
  19. company text,
  20. department text,
  21. number int,
  22. first_name text,
  23. last_name text,
  24. age int,
  25. married boolean,
  26. height bigint,
  27. weight float,
  28. birth_date timestamp,
  29. phones blob,
  30. primary key ((company, department), number)
  31. )
  32. with comment = 'A most excellent and useful table' AND read_repair_chance = 0.2 and clustering order by (number desc);

不要忘了设置正确的CLASSPATH环境变量:

  1. 在CLASSPATH中包含Ignite Cassandra模块的jar文件(ignite-cassandra-<version-number>.jar);
  2. 如果打算为部分自定义Java类使用POJO持久化策略,需要同时在CLASSPATH中包含带有这些类的jar文件。

6.5.负载测试

Ignite的Cassandra模块提供了一组负载测试,他可以模拟Ignite和Cassandra在自定义键/值类和持久化设定时的生产负载,因此使用这些负载测试就可以度量特定配置下的性能指标。

这种类型的度量有助于更好地理解系统的可度量性。

6.5.1.构建负载测试

Cassandra模块的负载测试是作为模块测试源代码的一部分提供的,因此首先需要从源代码构建Ignite发行版。
从源代码构建Ignite发行版之后,会发现在Cassandra模块目录下有target/tests-package目录以及target/ignite-cassandra-tests-<version>.zip,他是这个目录的zip压缩包。测试包包含了马上就可以用的Ignite Cassandra模块的负载测试应用,它的结构如下:

从上述列出的脚本可以看出,实际上有两种类型的负载测试:

下面看一下负载测试场景和配置细节。

6.5.2.负载测试场景

CassandraIgnite负载测试都使用相同的一组测试:

  1. 单个操作负载测试:当调用缓存的IgniteCache.put方法时就会执行这样的操作;
  2. 批量写操作负载测试:当调用缓存的IgniteCache.putAll方法时就会执行这样的操作;
  3. 单个操作负载测试:当调用缓存的IgniteCache.get方法时就会执行这样的操作;
  4. 批量读操作负载测试:当调用缓存的IgniteCache.getAll方法时就会执行这样的操作。

所有指定的负载测试都会按照给定的顺序一个一个按顺序执行。

6.5.3.负载测试配置

负载测试的配置是通过settings文件夹中的如下属性文件指定的:

  1. log4j.properties:指定负载测试的logger设置;
  2. org/apache/ignite/tests/cassandra/connection.properties:Cassandra连接的接触点设置(可以看下面的表格);
  3. org/apache/ignite/tests/cassandra/credentials.properties:Cassandra连接的凭证(可以看下面的表格);
  4. ests.properties:指定负载测试执行的配置信息(可以看下面的表格)。

log4j.properties文件非常简单,他就是log4j的logger配置,我们可以深入地了解下其他的配置文件。
org/apache/ignite/tests/cassandra/connection.properties

属性 描述
contact.points 用作接触点的Cassandra节点的列表,逗号分隔,应该是这样的格式:server-1[:port],server-2[:port],server-3[:port]等等

org/apache/ignite/tests/cassandra/credentials.properties

属性 描述
admin.user 连接Cassandra的Admin用户名
admin.password 连接Cassandra的Admin密码
regular.user 连接Cassandra的常规用户名
regular.password 连接Cassandra的常规用户密码

tests.properties

属性 描述
bulk.operation.size 每个批处理操作试图操作的元素数量:IgniteCache.getAll, IgniteCache.putAll
load.tests.cache.name 用于负载测试的Ignite缓存名
load.tests.threads.count 用于每个负载测试的线程数
load.tests.warmup.period 在开始任何度量前每个负载测试的预热周期(毫秒)。
load.tests.execution.time 除了预热周期外,每个负载测试的执行时间(毫秒)。
load.tests.requests.latency 到Cassandra/Ignite的两个连续请求的延迟时间(毫秒)。
load.tests.persistence.settings 用于负载测试的Cassandra持久化配置的资源
load.tests.ignite.config 用于负载测试的Ignite集群连接配置的资源
load.tests.key.generator Ignite缓存的键对象生成器
load.tests.value.generator Ignite缓存的值对象生成器

6.5.4.运行负载测试

运行负载测试前,要确保:

  1. Ignite缓存的所有节点都要配置为使用相同的Cassandra缓存存储配置。可以从测试源代码包的一个资源文件中找到一个远程Ignite节点配置的示例:settings/org/apache/ignite/tests/persistence/primitive/ignite-remote-server-config.xml,如果只打算使用Cassandra负载测试的话,可以忽略这一步;
  2. tests.propertiesload.tests.ignite.config属性,指向了正确的Ignite客户端节点配置(他应该有相同的Cassandra连接和持久化配置,比如远程Ignite节点配置文件),如果只打算使用Cassandra负载测试的话,可以忽略这一步;
  3. Cassandra连接配置,在org/apache/ignite/tests/cassandra/connection.propertiesorg/apache/ignite/tests/cassandra/credentials.properties文件中正确指定;

之后,只要执行相应的shell脚本,就可以轻易地运行负载测试了:

6.5.5.使用自定义的键/值类

如果使用自定义的键/值类用于负载测试,那么需要:

  1. tests.propertiesload.tests.key.generator属性中指定自定义键类的生成器。生成器的想法非常简单,他负责生成特定类的实例,并且实现org.apache.ignite.tests.load.Generator接口的public Object generate(long i)方法,也可以找到如下的示例实现:
    • org.apache.ignite.tests.load.IntGenerator:生成int实例;
    • rg.apache.ignite.tests.load.LongGenerator:生成long实例;
    • org.apache.ignite.tests.load.PersonIdGenerator:生成自定义Ignite缓存键类org.apache.ignite.tests.pojos.PersonId的实例;
    • org.apache.ignite.tests.load.PersonGenerator:生成自定义Ignite缓存值类org.apache.ignite.tests.pojos.Person的实例;
  2. tests.propertiesload.tests.value.generator属性中指定自定义值类的生成器。实际上他可能与前述提到的相同,但是他生成的是对应的自定义值对象而不是键对象;
  3. 对于自定义键/值类指定Cassandra持久化配置,将持久化配置文件放在settings文件夹,并且检查tests.properties中的load.tests.persistence.settings属性指向了自定义的持久化配置;
  4. 定义Ignite缓存配置使其使用自定义的持久化配置,并且检查tests.properties中的load.tests.ignite.config属性指向Ignite缓存配置文件;
  5. 创建包含自定义键/值类和对应生成器类的jar文件,然后将他们(还包括可能的第三方依赖)放在lib目录中。

这就是使用自定义类运行负载测试的所有步骤。

6.5.6.分析测试执行结果

负载测试的执行结果以log4j日志文件的形式提供,默认(如果没有修改log4j.properties文件)会有两个文件会显示测试执行的结果概要:

  1. cassandra-load-tests.log:包含了Cassandra负载测试执行结果的概要统计(如果执行的cassandra-load-tests.sh或者cassandra-load-tests.batshell脚本);
  2. ignite-load-tests.log:包含了Ignite负载测试执行结果的概要统计(如果执行的ignite-load-tests.sh或者ignite-load-tests.batshell脚本)。

下面是一个gnite-load-tests.log文件的示例(Cassandra负载测试的日志看上去差不多):

  1. 19:53:37,303 INFO [main] - Ignite load tests execution started
  2. 19:53:37,305 INFO [main] - Running WRITE test
  3. 19:53:37,305 INFO [main] - Setting up load tests driver
  4. 19:53:42,352 INFO [main] - Load tests driver setup successfully completed
  5. 19:53:42,355 INFO [main] - Starting workers
  6. 19:53:42,384 INFO [main] - Workers started
  7. 19:53:42,385 INFO [main] - Waiting for workers to complete
  8. 20:01:42,398 INFO [main] - Worker WRITE-worker-0 successfully completed
  9. 20:01:42,407 INFO [main] - Worker WRITE-worker-1 successfully completed
  10. 20:01:42,452 INFO [main] - Worker WRITE-worker-2 successfully completed
  11. 20:01:42,453 INFO [main] - Worker WRITE-worker-3 successfully completed
  12. 20:01:42,453 INFO [main] - Worker WRITE-worker-4 successfully completed
  13. 20:01:42,453 INFO [main] - Worker WRITE-worker-5 successfully completed
  14. 20:01:42,453 INFO [main] - Worker WRITE-worker-6 successfully completed
  15. 20:01:42,453 INFO [main] - Worker WRITE-worker-7 successfully completed
  16. 20:01:42,453 INFO [main] - Worker WRITE-worker-8 successfully completed
  17. 20:01:42,453 INFO [main] - Worker WRITE-worker-9 successfully completed
  18. 20:01:42,453 INFO [main] - Worker WRITE-worker-10 successfully completed
  19. 20:01:42,453 INFO [main] - Worker WRITE-worker-11 successfully completed
  20. 20:01:42,453 INFO [main] - Worker WRITE-worker-12 successfully completed
  21. 20:01:42,453 INFO [main] - Worker WRITE-worker-13 successfully completed
  22. 20:01:42,453 INFO [main] - Worker WRITE-worker-14 successfully completed
  23. 20:01:42,453 INFO [main] - Worker WRITE-worker-15 successfully completed
  24. 20:01:42,454 INFO [main] - Worker WRITE-worker-16 successfully completed
  25. 20:01:42,454 INFO [main] - Worker WRITE-worker-17 successfully completed
  26. 20:01:42,639 INFO [main] - Worker WRITE-worker-18 successfully completed
  27. 20:01:42,639 INFO [main] - Worker WRITE-worker-19 successfully completed
  28. 20:01:42,639 INFO [main] - WRITE test execution successfully completed.
  29. 20:01:42,639 INFO [main] -
  30. -------------------------------------------------
  31. WRITE test statistics
  32. WRITE messages: 1681780
  33. WRITE errors: 0, 0.00%
  34. WRITE speed: 5597 msg/sec
  35. -------------------------------------------------
  36. 20:01:42,695 INFO [main] - Running BULK_WRITE test
  37. 20:01:42,695 INFO [main] - Setting up load tests driver
  38. 20:01:45,074 INFO [main] - Load tests driver setup successfully completed
  39. 20:01:45,074 INFO [main] - Starting workers
  40. 20:01:45,093 INFO [main] - Workers started
  41. 20:01:45,094 INFO [main] - Waiting for workers to complete
  42. 20:09:45,084 INFO [main] - Worker BULK_WRITE-worker-0 successfully completed
  43. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-1 successfully completed
  44. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-2 successfully completed
  45. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-3 successfully completed
  46. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-4 successfully completed
  47. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-5 successfully completed
  48. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-6 successfully completed
  49. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-7 successfully completed
  50. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-8 successfully completed
  51. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-9 successfully completed
  52. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-10 successfully completed
  53. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-11 successfully completed
  54. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-12 successfully completed
  55. 20:09:45,105 INFO [main] - Worker BULK_WRITE-worker-13 successfully completed
  56. 20:09:45,254 INFO [main] - Worker BULK_WRITE-worker-14 successfully completed
  57. 20:09:45,254 INFO [main] - Worker BULK_WRITE-worker-15 successfully completed
  58. 20:09:45,254 INFO [main] - Worker BULK_WRITE-worker-16 successfully completed
  59. 20:09:45,258 INFO [main] - Worker BULK_WRITE-worker-17 successfully completed
  60. 20:09:45,258 INFO [main] - Worker BULK_WRITE-worker-18 successfully completed
  61. 20:09:45,258 INFO [main] - Worker BULK_WRITE-worker-19 successfully completed
  62. 20:09:45,258 INFO [main] - BULK_WRITE test execution successfully completed.
  63. 20:09:45,258 INFO [main] -
  64. -------------------------------------------------
  65. BULK_WRITE test statistics
  66. BULK_WRITE messages: 2021500
  67. BULK_WRITE errors: 0, 0.00%
  68. BULK_WRITE speed: 6748 msg/sec
  69. -------------------------------------------------
  70. 20:09:45,477 INFO [main] - Running READ test
  71. 20:09:45,477 INFO [main] - Setting up load tests driver
  72. 20:09:48,626 INFO [main] - Load tests driver setup successfully completed
  73. 20:09:48,626 INFO [main] - Starting workers
  74. 20:09:48,631 INFO [main] - Workers started
  75. 20:09:48,631 INFO [main] - Waiting for workers to complete
  76. 20:17:57,128 INFO [main] - Worker READ-worker-0 successfully completed
  77. 20:17:57,128 INFO [main] - Worker READ-worker-1 successfully completed
  78. 20:17:57,128 INFO [main] - Worker READ-worker-2 successfully completed
  79. 20:17:57,128 INFO [main] - Worker READ-worker-3 successfully completed
  80. 20:17:57,145 INFO [main] - Worker READ-worker-4 successfully completed
  81. 20:17:57,145 INFO [main] - Worker READ-worker-5 successfully completed
  82. 20:17:57,145 INFO [main] - Worker READ-worker-6 successfully completed
  83. 20:17:57,145 INFO [main] - Worker READ-worker-7 successfully completed
  84. 20:17:57,145 INFO [main] - Worker READ-worker-8 successfully completed
  85. 20:17:57,145 INFO [main] - Worker READ-worker-9 successfully completed
  86. 20:17:57,145 INFO [main] - Worker READ-worker-10 successfully completed
  87. 20:17:57,145 INFO [main] - Worker READ-worker-11 successfully completed
  88. 20:17:57,145 INFO [main] - Worker READ-worker-12 successfully completed
  89. 20:17:57,145 INFO [main] - Worker READ-worker-13 successfully completed
  90. 20:17:57,145 INFO [main] - Worker READ-worker-14 successfully completed
  91. 20:17:57,145 INFO [main] - Worker READ-worker-15 successfully completed
  92. 20:17:57,145 INFO [main] - Worker READ-worker-16 successfully completed
  93. 20:17:57,145 INFO [main] - Worker READ-worker-17 successfully completed
  94. 20:17:57,145 INFO [main] - Worker READ-worker-18 successfully completed
  95. 20:17:57,145 INFO [main] - Worker READ-worker-19 successfully completed
  96. 20:17:57,145 INFO [main] - READ test execution successfully completed.
  97. 20:17:57,145 INFO [main] -
  98. -------------------------------------------------
  99. READ test statistics
  100. READ messages: 1974957
  101. READ errors: 0, 0.00%
  102. READ speed: 6404 msg/sec
  103. -------------------------------------------------
  104. 20:17:57,207 INFO [main] - Running BULK_READ test
  105. 20:17:57,207 INFO [main] - Setting up load tests driver
  106. 20:17:59,495 INFO [main] - Load tests driver setup successfully completed
  107. 20:17:59,495 INFO [main] - Starting workers
  108. 20:17:59,515 INFO [main] - Workers started
  109. 20:17:59,515 INFO [main] - Waiting for workers to complete
  110. 20:25:59,568 INFO [main] - Worker BULK_READ-worker-0 successfully completed
  111. 20:25:59,568 INFO [main] - Worker BULK_READ-worker-1 successfully completed
  112. 20:25:59,568 INFO [main] - Worker BULK_READ-worker-2 successfully completed
  113. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-3 successfully completed
  114. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-4 successfully completed
  115. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-5 successfully completed
  116. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-6 successfully completed
  117. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-7 successfully completed
  118. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-8 successfully completed
  119. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-9 successfully completed
  120. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-10 successfully completed
  121. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-11 successfully completed
  122. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-12 successfully completed
  123. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-13 successfully completed
  124. 20:25:59,585 INFO [main] - Worker BULK_READ-worker-14 successfully completed
  125. 20:25:59,586 INFO [main] - Worker BULK_READ-worker-15 successfully completed
  126. 20:25:59,586 INFO [main] - Worker BULK_READ-worker-16 successfully completed
  127. 20:25:59,586 INFO [main] - Worker BULK_READ-worker-17 successfully completed
  128. 20:25:59,586 INFO [main] - Worker BULK_READ-worker-18 successfully completed
  129. 20:25:59,586 INFO [main] - Worker BULK_READ-worker-19 successfully completed
  130. 20:25:59,586 INFO [main] - BULK_READ test execution successfully completed.
  131. 20:25:59,586 INFO [main] -
  132. -------------------------------------------------
  133. BULK_READ test statistics
  134. BULK_READ messages: 3832300
  135. BULK_READ errors: 0, 0.00%
  136. BULK_READ speed: 12790 msg/sec
  137. -------------------------------------------------
  138. 20:25:59,653 INFO [main] - Ignite load tests execution completed

根据这个日志,可以发现:

因此要为集群模拟真实的负载,只需要从多个客户端节点同时运行同一个负载测试,然后统计所有节点每个测试的平均速度,这就是当前的配置能处理的写/批量写/读/批量读操作的平均速度。

如果打算使用AWS进行负载测试,只需要使用AWS部署,他会自动地考虑所有的路由(创建和引导必要的用于Ignite/Cassandra/Tests集群的EC2实例,运行负载测试以及等待他们完成,从每个EC2实例中收集所有的负载测试统计并且产生总结报告)。作为额外的好处,还可以得到Cassandra/Ignite/Tests集群的基于Ganglia的监控,他可以看到集群在高负载时都发生了什么。

建议运行负载测试之前执行recreate-cassandra-artifacts.sh/recreate-cassandra-artifacts.bat脚本。这个脚本会清理之前的负载测试执行中产生的所有Cassandra键空间/表,否则统计可能不是很精确,如果使用AWS,这个会自动做。

6.6.单元测试

Ignite的Cassandra模块提供了一组单元测试用例,它可以用来测试下面的功能是否正常:

  1. 绕过Ignite,通过直接序列化/反序列化Java基本类型(int,float,long等)到Cassandra测试PRIMITIVE持久化策略;
  2. 绕过Ignite,使用Java和Kryo序列化系统,通过直接序列化/反序列化Java基本类型和自定义Java类型到Cassandra测试BLOB持久化策略;
  3. 绕过Ignite,通过直接序列化/反序列化自定义Java类型到Cassandra测试POJO持久化策略;
  4. 透过Ignite缓存测试PRIMITIVE持久化策略;
  5. 透过Ignite缓存测试BLOB持久化策略;
  6. 透过Ignite缓存测试POJO持久化策略;
  7. 通过loadCache操作预热Ignite缓存。
添加新批注
在作者公开此批注前,只有你和作者可见。
回复批注