créer une base nosql en 1 heure
DESCRIPTION
Conférence données à l'Open World Forum, 05 octobre 2013. Comment créer une base de données noSQL par paires clés-valeurs en moins d'une heure, en se basant sur le bibliothèques Nanomsg et LightningDB.TRANSCRIPT
![Page 1: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/1.jpg)
Créer un serveur noSQLen une heure
![Page 2: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/2.jpg)
Bases noSQL
Bibliothèques utilisées
Présentation AngstromDB
Étude de code
Benchmark & évolutions
Créer un serveur noSQL
![Page 3: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/3.jpg)
Base noSQL
![Page 4: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/4.jpg)
Aller plus loin
Scalabilité horizontale
Haute disponibilité
Big Data
S'affranchir
Modèle relationnel
Transactions ACID
Principe général
![Page 5: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/5.jpg)
Types de bases noSQL
Clé-valeur Document Colonne GraphDynamo
Riak
Redis
Voldemort
Tokyo Tyrant
MemcacheDB
FoundationDB
MongoDB
CouchDB
CouchBase
HyperDex
RethinkDB
Cassandra
Hadoop / Hbase
Accumulo
Neo4J
Allegro
Virtuoso
InfoGrid
![Page 6: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/6.jpg)
Bibliothèquesde programmation
![Page 7: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/7.jpg)
Le successeur de ZeroMQ
Bibliothèque réseau avec des super-pouvoirs
Nanomsg
![Page 8: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/8.jpg)
Gestion de files de messages
Protocole spécifique
14 langages supportés
Redéfinition des concepts réseau
Nanomsg
![Page 9: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/9.jpg)
Méthodes de transport
Inter-threads inproc
Inter-processus ipc
Inter-machines tcp
![Page 10: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/10.jpg)
Découplage du sens de connexion
ServeurClient
![Page 11: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/11.jpg)
Découplage du sens de connexion
ServeurClient
![Page 12: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/12.jpg)
Socket multi-connexion
ServeurClient
Client
Client
port A
port B
![Page 13: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/13.jpg)
Connexion REQ/REP
ServeurClientREQ
![Page 14: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/14.jpg)
Connexion REQ/REP
ServeurClientREP
![Page 15: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/15.jpg)
Connexion PUSH/PULL
ServeurClientPUSH
![Page 16: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/16.jpg)
Connexion PUSH/PULL
ServeurClientPULL
![Page 17: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/17.jpg)
Connexion PUB/SUB
ClientPUB
Serveur
Client
Client
![Page 18: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/18.jpg)
Connexion PUB/SUB
ClientServeur
Client
Client
SUB
SUB
SUB
![Page 19: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/19.jpg)
Load-balancing
ClientServeur
Client
Client
PULL
PUSH
![Page 20: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/20.jpg)
Load-balancing
ClientServeur
Client
Client
PULLPUSH
![Page 21: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/21.jpg)
Load-balancing
ClientServeur
Client
ClientPULL
PUSH
![Page 22: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/22.jpg)
Bus
Client
Client
Client
Client
![Page 23: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/23.jpg)
Bus
Client Client
Client
Client
![Page 24: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/24.jpg)
Stockage paires clé-valeur multivaluées
Persistance sur disque, mapping RAM
Transactionnel (1 thread d'écriture)
Au cœur de OpenLDAP
LMDB (LightningDB)
![Page 25: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/25.jpg)
Débit
Latence
ops / sec.
microsec.
HyperDex : LMDB vs LevelDB
![Page 26: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/26.jpg)
PrésentationAngstromDB
![Page 27: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/27.jpg)
Serveur paires clé-valeur
Codé en C
Basé sur des bibliothèques reconnues
Protocole binaire
Aucun contrôle d'erreur
Implémentation naïve
![Page 28: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/28.jpg)
API très simple
PUT Ajout ou mise-à-jour de clé
GET Retourne une valeur
DELETE Efface une clé
![Page 29: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/29.jpg)
Protocole : DELETE
2
cmd taille clé
clé
1 1 1 à 255 octets
Requête
Réponse 1
statut
1
![Page 30: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/30.jpg)
Protocole : GET
3
cmd taille clé
clé
1 1 1 à 255 octets
Requête
Réponse 1
statut taille données
données
1 4 0 à 4 GO
![Page 31: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/31.jpg)
Protocole : PUT
1
cmd taille clé clé
1 1 1 à 255 octets
Requête
Réponse 1
statut
taille données données
1
4 0 à 4 GO
![Page 32: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/32.jpg)
Multi-threadé
Thread principal Attend les nouvelles connexions
Threads de communication Un thread par client connecté
Thread d'écriture Pas d'écriture concurrente
![Page 33: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/33.jpg)
Thread principal
Threads de communication
LMDBThread
d'écriture
![Page 34: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/34.jpg)
Étude de code
![Page 35: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/35.jpg)
github.com/Amaury/AngstromDB ↩
↪ /tree/master/src
![Page 36: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/36.jpg)
Démarrage
/*
* db Pointer to the database environment.
* socket Socket descriptor for incoming connections.
* threads_socket Nanomsg socket for threads communication.
* writer_tid ID of the writer thread.
* comm_threads Array of communication threads.
*/
typedef struct angstrom_s {
MDB_env *db;
int socket;
int threads_socket;
pthread_t writer_tid;
struct comm_thread_s *comm_threads;
} angstrom_t;
/*
* tid Thread's identifier.
* angstrom Pointer to the server's structure.
* client_sock Socket used to communicate with the client.
* writer_sock Nanomsg socket to send data to the writer. */
typedef struct comm_thread_s {
pthread_t tid;
angstrom_t *angstrom;
int client_sock;
int writer_sock;
} comm_thread_t;
angstrom.h
![Page 37: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/37.jpg)
Démarrage
int main() {
angstrom_t *angstrom;
int i;
// server init
angstrom = calloc(1, sizeof(angstrom_t));
angstrom->socket = angstrom->threads_socket = -1;
angstrom->comm_threads = calloc(NBR_THREADS,
sizeof(comm_thread_t));
// open the database
angstrom->db = database_open(DEFAULT_DB_PATH,
DEFAULT_MAPSIZE, NBR_THREADS);
// create the nanomsg socket for threads communication
angstrom->threads_socket = nn_socket(AF_SP, NN_PUSH);
nn_bind(angstrom->threads_socket, ENDPOINT_THREADS_SOCKET);
// create the writer thread
pthread_create(&angstrom->writer_tid, NULL,
thread_writer_loop, angstrom);
main.c
![Page 38: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/38.jpg)
Démarrage
int main() {
angstrom_t *angstrom;
int i;
// server init
angstrom = calloc(1, sizeof(angstrom_t));
angstrom->socket = angstrom->threads_socket = -1;
angstrom->comm_threads = calloc(NBR_THREADS,
sizeof(comm_thread_t));
// open the database
angstrom->db = database_open(DEFAULT_DB_PATH,
DEFAULT_MAPSIZE, NBR_THREADS);
// create the nanomsg socket for threads communication
angstrom->threads_socket = nn_socket(AF_SP, NN_PUSH);
nn_bind(angstrom->threads_socket, ENDPOINT_THREADS_SOCKET);
// create the writer thread
pthread_create(&angstrom->writer_tid, NULL,
thread_writer_loop, angstrom);
main.c
"inproc://threads_socket"
![Page 39: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/39.jpg)
Démarrage
// create communication threads
for (i = 0; i < NBR_THREADS; i++) {
comm_thread_t *thread = &(angstrom->comm_threads[i]);
thread->client_sock = -1;
thread->angstrom = angstrom;
pthread_create(&thread->tid, 0, thread_comm_loop,
thread);
pthread_detach(thread->tid);
}
// create listening socket
angstrom->socket = _create_listening_socket(DEFAULT_PORT);
// server loop
_main_thread_loop(angstrom);
return (0);
}
main.c
![Page 40: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/40.jpg)
Démarrage
MDB_env *database_open(const char *path, size_t mapsize,
unsigned int nbr_threads) {
MDB_env *env = NULL;
mdb_env_create(&env);
mdb_env_set_mapsize(env, mapsize);
mdb_env_set_maxreaders(env, nbr_threads);
mdb_env_open(env, path, 0, 0664);
return (env);
}
database.c
![Page 41: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/41.jpg)
Démarrage
static int _create_listening_socket(unsigned short port) {
int sock;
struct sockaddr_in addr;
unsigned int addr_size;
const int on = 1;
// create the socket
sock = socket(AF_INET, SOCK_STREAM, 0);
// some options
setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, (void*)&on,
sizeof(on));
setsockopt(sock, SOL_SOCKET, SO_KEEPALIVE, (void*)&on,
sizeof(on));
// binding to any interface
addr_size = sizeof(addr);
bzero(&addr, addr_size);
addr.sin_addr.s_addr = htonl(INADDR_ANY);
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
bind(sock, (struct sockaddr*)&addr, addr_size);
listen(sock, SOMAXCONN);
return (sock);
}
main.c
![Page 42: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/42.jpg)
Nouvelle connexion
static void _main_thread_loop(angstrom_t *angstrom) {
int fd;
struct sockaddr_in addr;
unsigned int addr_size;
const int on = 1;
addr_size = sizeof(addr);
for (; ; ) {
bzero(&addr, addr_size);
// accept a new connection
if ((fd = accept(angstrom->socket,
(struct sockaddr*)&addr,
&addr_size)) < 0) {
continue ;
}
setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, (void*)&on,
sizeof(on));
// send the file descriptor number to comm threads
nn_send(angstrom->threads_socket, &fd, sizeof(fd), 0);
}
}
main.c
![Page 43: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/43.jpg)
Nouvelle connexion
void *thread_comm_loop(void *param) {
comm_thread_t *thread = param;
int in_sock;
// opening a connection to the writer thread
thread->writer_sock = nn_socket(AF_SP, NN_PUSH);
nn_connect(thread->writer_sock, ENDPOINT_WRITER_SOCKET);
// opening a connection to the main thread
in_sock = nn_socket(AF_SP, NN_PULL);
nn_connect(in_sock, ENDPOINT_THREADS_SOCKET);
// loop to process new connections
for (; ; ) {
// waiting for a new connection to handle
nn_recv(in_sock, &thread->client_sock, sizeof(int), 0);
// process connection
_process_connection(thread);
}
return (NULL);
}
thread_communication.c
![Page 44: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/44.jpg)
Nouvelle connexion
void *thread_comm_loop(void *param) {
comm_thread_t *thread = param;
int in_sock;
// opening a connection to the writer thread
thread->writer_sock = nn_socket(AF_SP, NN_PUSH);
nn_connect(thread->writer_sock, ENDPOINT_WRITER_SOCKET);
// opening a connection to the main thread
in_sock = nn_socket(AF_SP, NN_PULL);
nn_connect(in_sock, ENDPOINT_THREADS_SOCKET);
// loop to process new connections
for (; ; ) {
// waiting for a new connection to handle
nn_recv(in_sock, &thread->client_sock, sizeof(int), 0);
// process connection
_process_connection(thread);
}
return (NULL);
}
thread_communication.c
"inproc://writer_socket"
"inproc://threads_socket"
![Page 45: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/45.jpg)
Nouvelle connexion
static void _process_connection(comm_thread_t *thread) {
uint8_t cmd;
// loop on incoming requests
for (; ; ) {
// read command byte
if (read(thread->client_sock, &cmd, sizeof(cmd)) <= 0) {
close(thread->client_sock);
break;
}
// interpret command
switch (cmd) {
case PROTO_PUT:
command_put(thread);
break;
case PROTO_DELETE:
command_del(thread);
break;
case PROTO_GET:
command_get(thread);
break;
}
}
}
thread_communication.c
![Page 46: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/46.jpg)
Lecture de données
void command_get(comm_thread_t *thread) {
uint8_t key_size, response = PROTO_OK;
uint32_t value_size;
MDB_val key, value;
// read key length
read(thread->client_sock, &key_size, sizeof(key_size));
// read key data
key.mv_data = malloc(key_size);
read(thread->client_sock, key.mv_data, key_size);
// get data
key.mv_size = (size_t)key_size;
database_get(thread->angstrom->db, &key, &value);
// send response to the client
write(thread->client_sock, &response, sizeof(response));
value_size = htonl((uint32_t)value.mv_size);
write(thread->client_sock, &value_size, sizeof(value_size));
if (value_size)
write(thread->client_sock, value.mv_data, value.mv_size);
}
command_get.c
![Page 47: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/47.jpg)
Lecture de données
void database_get(MDB_env *db, MDB_val *key, MDB_val *value) {
MDB_dbi dbi;
MDB_txn *txn;
// transaction init
mdb_txn_begin(db, NULL, MDB_RDONLY, &txn);
// open database in read-write mode
mdb_dbi_open(txn, NULL, 0, &dbi);
// get data
if (mdb_get(txn, dbi, key, value))
bzero(value, sizeof(*value));
// end of transaction
mdb_txn_abort(txn);
// close database
mdb_dbi_close(db, dbi);
}
database.c
![Page 48: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/48.jpg)
Thread d'écriture
/*
* type Type of action (WRITE_PUT, WRITE_DEL).
* key Size and content of the key.
* value Size and content of the value.
*/
typedef struct writer_msg_s {
enum {
WRITE_PUT,
WRITE_DEL
} type;
MDB_val key;
MDB_val value;
} writer_msg_t;
angstrom.h
![Page 49: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/49.jpg)
Thread d'écriture
void *thread_writer_loop(void *param) {
angstrom_t *angstrom = param;
int socket;
// create the nanomsg socket for threads communication
socket = nn_socket(AF_SP, NN_PULL);
nn_bind(socket, ENDPOINT_WRITER_SOCKET);
// loop to process new connections
for (; ; ) {
writer_msg_t *msg;
// waiting for a new connection to handle
if (nn_recv(socket, &msg, sizeof(writer_msg_t*), 0) < 0)
continue;
// processing
switch (msg->type) {
case WRITE_PUT:
database_put(angstrom->db, &msg->key, &msg->value);
break;
case WRITE_DEL:
database_del(angstrom->db, &msg->key);
break;
}
thread_writer.c
![Page 50: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/50.jpg)
Thread d'écriture
void *thread_writer_loop(void *param) {
angstrom_t *angstrom = param;
int socket;
// create the nanomsg socket for threads communication
socket = nn_socket(AF_SP, NN_PULL);
nn_bind(socket, ENDPOINT_WRITER_SOCKET);
// loop to process new connections
for (; ; ) {
writer_msg_t *msg;
// waiting for a new connection to handle
if (nn_recv(socket, &msg, sizeof(writer_msg_t*), 0) < 0)
continue;
// processing
switch (msg->type) {
case WRITE_PUT:
database_put(angstrom->db, &msg->key, &msg->value);
break;
case WRITE_DEL:
database_del(angstrom->db, &msg->key);
break;
}
thread_writer.c
"inproc://writer_socket"
![Page 51: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/51.jpg)
Thread d'écriture
// free data
if (msg->key.mv_data)
free(msg->key.mv_data);
if (msg->value.mv_data)
free(msg->value.mv_data);
free(msg);
}
return (NULL);
}
thread_writer.c
![Page 52: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/52.jpg)
Effacement de clé
void command_delete(comm_thread_t *thread) {
uint8_t key_size, response = PROTO_OK;
void *key;
writer_msg_t *msg;
// read key length
read(thread->client_sock, &key_size, sizeof(key_size));
// read key data
key = malloc(key_size);
read(thread->client_sock, key, key_size);
// create message
msg = calloc(1, sizeof(writer_msg_t));
msg->type = WRITE_DEL;
msg->key.mv_size = (size_t)key_size;
msg->key.mv_data = key;
// send the message to the writer thread
nn_send(thread->writer_sock, &msg, sizeof(msg), 0);
// send response to the client
write(thread->client_sock, &response, sizeof(response));
}
command_delete.c
![Page 53: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/53.jpg)
Effacement de clé
void database_del(MDB_env *db, MDB_val *key) {
MDB_dbi dbi;
MDB_txn *txn;
// transaction init
mdb_txn_begin(db, NULL, 0, &txn);
// open database in read-write mode
mdb_dbi_open(txn, NULL, 0, &dbi);
// delete key
mdb_del(txn, dbi, key, NULL);
// close database
mdb_dbi_close(db, dbi);
// transaction commit
mdb_txn_commit(txn);
}
database.c
![Page 54: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/54.jpg)
Ajout / mise-à-jour de clé
void command_put(comm_thread_t *thread) {
uint8_t key_size, response = PROTO_OK;
void *key, *value = NULL;
uint32_t value_size;
writer_msg_t *msg;
// read key length
read(thread->client_sock, &key_size, sizeof(key_size));
// read key data
key = malloc(key_size);
read(thread->client_sock, key, key_size);
// read value length
read(thread->client_sock, &value_size, sizeof(value_size));
value_size = ntohl(value_size);
if (value_size > 0) {
// read value data
value = malloc(value_size);
read(thread->client_sock, value, value_size);
}
command_put.c
![Page 55: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/55.jpg)
Ajout / mise-à-jour de clé
// create message
msg = malloc(sizeof(writer_msg_t));
msg->type = WRITE_PUT;
msg->key.mv_size = (size_t)key_size;
msg->key.mv_data = key;
msg->value.mv_size = (size_t)value_size;
msg->value.mv_data = value;
// send the message to the writer thread
nn_send(thread->writer_sock, &msg, sizeof(msg), 0);
// send response to the client
write(thread->client_sock, &response, sizeof(response));
}
command_put.c
![Page 56: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/56.jpg)
Ajout / mise-à-jour de clé
void database_put(MDB_env *db, MDB_val *key, MDB_val *value) {
MDB_dbi dbi;
MDB_txn *txn;
// transaction init
mdb_txn_begin(db, NULL, 0, &txn);
// open database in read-write mode
mdb_dbi_open(txn, NULL, 0, &dbi);
// put data
mdb_put(txn, dbi, key, value, 0);
// close database
mdb_dbi_close(db, dbi);
// transaction commit
mdb_txn_commit(txn);
}
database.c
![Page 57: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/57.jpg)
![Page 58: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/58.jpg)
clocLanguage files comment code-------------------------------------------------C 7 132 212C Header 1 111 55make 1 13 27-------------------------------------------------SUM: 9 256 294
sloccountSchedule Estimate, Years (Months) = 0.17 (2.06)
Total Estimated Cost to Develop = $ 6,753
![Page 59: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/59.jpg)
Benchmark
Base Écritures Lectures
AngstromDB 22 ms 55 ms
Couchbase 25 ms 22 ms
Redis 29 ms 30 ms
MongoDB 39 ms 27 ms
45 écritures / lectures séquentielles de données représentatives
![Page 60: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/60.jpg)
Améliorationspossibles
![Page 61: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/61.jpg)
Réduire les appels systèmes !
Bufferiser les lectures réseau
Regrouper les écritures réseau
Facile à tester (sendmsg vs write)
Pour commencer
![Page 62: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/62.jpg)
Benchmark
Base Écritures Lectures
AngstromDB 22 ms 55 ms
AngstromDB 22 ms 26 ms
Couchbase 25 ms 22 ms
Redis 29 ms 30 ms
MongoDB 39 ms 27 ms
![Page 63: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/63.jpg)
Compression à la volée (Zippy)
Sérialisation de données (MsgPack)
Transactions en lecture
… Mono-processus asynchrone ?
Pour aller plus loin
![Page 64: Créer une base NoSQL en 1 heure](https://reader033.vdocuments.site/reader033/viewer/2022042502/5558231ed8b42a5e468b50ad/html5/thumbnails/64.jpg)
geek-directeur-technique.com
github.com/Amaury/AngstromDB
@geekcto