Category Archives: Administration système

Monitoring-FR : deux articles sur Centreon et Ansible

Je continue d’écrire des articles pour Monitoring-fr. Cette fois-ci, j’ai écrit deux articles sur Centreon et Ansible :

  1. Automatisation de la supervision : exemple avec Centreon et Ansible (1)
  2. Automatisation de la supervision : exemple avec Centreon et Ansible (2)

Le sujet n’est pas tant Centreon ou Ansible mais « l’automatisation de la supervision ». Les logiciels sont uniquemement cités pour illustrer les articles.

Monitoring-FR : Shinken 2.4

 

J’ai écrit un article sur Monitoring-FR sur la sortie de la version 2.4 de Shinken. Pour écrire cet article, j’ai interviewé Jean « Naparuba » Gabes par téléphone, le fondateur de Shinken et son principal développeur. Je connais moins bien Shinken que d’autres outils de supervision comme Centreon, Nagios ou Cacti. J’espère ne pas avoir écrit des énormités 🙂

Je vous invite à lire l’article et à le commenter directement sur le site Monitoring-FR : la sortie de la version 2.4 de Shinken.

Valider les évolutions d’une API JSON

 

Vous exposez des données au format JSON au travers d’une API. Cette API est utilisée par d’autres personnes extérieures à votre structure. Vous êtes responsables des informations fournies à ces personnes extérieures, vous devez les assister lors de leur première utilisation en répondant à leurs questions mais aussi lors des mises à jour. Bien entendu, la documentation de l’API a été demandée aux développeurs de cette API! Évidemment, cette documentation semble avoir été faite! Assurément, la liste des différences entre l’ancienne version et la nouvelle version est détaillée! Incontestablement, cette liste de différences est complète, claire et exhaustive! Il n’y a donc aucun problème! Jamais! D’ailleurs, aucune modification n’est faite dans l’API entre les versions mineures. D’autre part, le projet n’est jamais en retard et la livraison est faite systématiquement 2 semaines avant la mise en production afin de vous donner le temps de tout vérifier. Durant cette phase de vérification, vous n’êtes jamais perturbé par d’autres tâches annexes. Vous n’avez jamais constaté un bug imposant une nouvelle livraison, afin changement de cette API, dans une partie non prévue. Si vous ne connaissez pas ce monde de bisounours (si vous le connaissez, dites-moi afin que j’y émigre sans délai), il vous faut valider et vérifier chaque point. Ce qui peut être long lorsque ces modifications sont au nombre de 285. Comment automatiser un peu la vérification et avoir une liste synthétique des différences? Comment être plus efficace pour vérifier une API fournissant des données au format JSON? C’est ce que nous allons voir ici.

Continue reading Valider les évolutions d’une API JSON

Jouons avec JSON

 

Le très gros point fort du JSON est qu’il est très peu verbeux (contrairement au XML) et performant à analyser (contrairement au …). C’est aussi un point faible pour nous administrateurs, humains, qui avons du mal à le comprendre. Pour cela, il faut pouvoir le lire. Ce n’est pas toujours évident, comme le montre cet exemple :

{"_shards":{"total":90,"successful":45,"failed":0},"_all":{"primaries":{"docs":{"count":99254,"deleted":0},"store":{"size_in_bytes":192367723,"throttle_time_in_millis":28623},"indexing":{"index_total":99254,"index_time_in_millis":218667,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":24550,"time_in_millis":2000,"exists_total":24534,"exists_time_in_millis":2000,"missing_total":16,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":993640,"query_time_in_millis":4358463,"query_current":0,"fetch_total":197801,"fetch_time_in_millis":40594,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":773,"total_time_in_millis":43183},"flush":{"total":50,"total_time_in_millis":12412},"warmer":{"current":0,"total":822,"total_time_in_millis":1275},"filter_cache":{"memory_size_in_bytes":3495180,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":32400201,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":238,"memory_in_bytes":1600228},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":99254,"deleted":0},"store":{"size_in_bytes":192367723,"throttle_time_in_millis":28623},"indexing":{"index_total":99254,"index_time_in_millis":218667,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":24550,"time_in_millis":2000,"exists_total":24534,"exists_time_in_millis":2000,"missing_total":16,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":993640,"query_time_in_millis":4358463,"query_current":0,"fetch_total":197801,"fetch_time_in_millis":40594,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":773,"total_time_in_millis":43183},"flush":{"total":50,"total_time_in_millis":12412},"warmer":{"current":0,"total":822,"total_time_in_millis":1275},"filter_cache":{"memory_size_in_bytes":3495180,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":32400201,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":238,"memory_in_bytes":1600228},"translog":{"operations":0,"size_in_bytes":0}}},"indices":{"vm-dev_123_2014-10-16_19-04-medias":{"primaries":{"docs":{"count":43053,"deleted":0},"store":{"size_in_bytes":9326582,"throttle_time_in_millis":3059},"indexing":{"index_total":43053,"index_time_in_millis":13411,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":199,"total_time_in_millis":2249},"flush":{"total":10,"total_time_in_millis":5592},"warmer":{"current":0,"total":220,"total_time_in_millis":119},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":15,"memory_in_bytes":338520},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":43053,"deleted":0},"store":{"size_in_bytes":9326582,"throttle_time_in_millis":3059},"indexing":{"index_total":43053,"index_time_in_millis":13411,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":199,"total_time_in_millis":2249},"flush":{"total":10,"total_time_in_millis":5592},"warmer":{"current":0,"total":220,"total_time_in_millis":119},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":15,"memory_in_bytes":338520},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_16_2014-10-21_12-17-artworks":{"primaries":{"docs":{"count":2348,"deleted":0},"store":{"size_in_bytes":9316186,"throttle_time_in_millis":0},"indexing":{"index_total":2348,"index_time_in_millis":7756,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":4702,"time_in_millis":678,"exists_total":4686,"exists_time_in_millis":678,"missing_total":16,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":407110,"query_time_in_millis":429377,"query_current":0,"fetch_total":106584,"fetch_time_in_millis":13256,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":32,"total_time_in_millis":731},"flush":{"total":5,"total_time_in_millis":219},"warmer":{"current":0,"total":37,"total_time_in_millis":13},"filter_cache":{"memory_size_in_bytes":99792,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":4375465,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":27,"memory_in_bytes":63977},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":2348,"deleted":0},"store":{"size_in_bytes":9316186,"throttle_time_in_millis":0},"indexing":{"index_total":2348,"index_time_in_millis":7756,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":4702,"time_in_millis":678,"exists_total":4686,"exists_time_in_millis":678,"missing_total":16,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":407110,"query_time_in_millis":429377,"query_current":0,"fetch_total":106584,"fetch_time_in_millis":13256,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":32,"total_time_in_millis":731},"flush":{"total":5,"total_time_in_millis":219},"warmer":{"current":0,"total":37,"total_time_in_millis":13},"filter_cache":{"memory_size_in_bytes":99792,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":4375465,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":27,"memory_in_bytes":63977},"translog":{"operations":0,"size_in_bytes":0}}},"vm-stg_29_2014-10-23_17-34-artworks":{"primaries":{"docs":{"count":1839,"deleted":0},"store":{"size_in_bytes":6950220,"throttle_time_in_millis":0},"indexing":{"index_total":1839,"index_time_in_millis":26329,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":857,"time_in_millis":36,"exists_total":857,"exists_time_in_millis":36,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":54700,"query_time_in_millis":48757,"query_current":0,"fetch_total":3876,"fetch_time_in_millis":395,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":28,"total_time_in_millis":7022},"flush":{"total":5,"total_time_in_millis":374},"warmer":{"current":0,"total":33,"total_time_in_millis":73},"filter_cache":{"memory_size_in_bytes":100968,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":4120891,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":23,"memory_in_bytes":53502},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":1839,"deleted":0},"store":{"size_in_bytes":6950220,"throttle_time_in_millis":0},"indexing":{"index_total":1839,"index_time_in_millis":26329,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":857,"time_in_millis":36,"exists_total":857,"exists_time_in_millis":36,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":54700,"query_time_in_millis":48757,"query_current":0,"fetch_total":3876,"fetch_time_in_millis":395,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":28,"total_time_in_millis":7022},"flush":{"total":5,"total_time_in_millis":374},"warmer":{"current":0,"total":33,"total_time_in_millis":73},"filter_cache":{"memory_size_in_bytes":100968,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":4120891,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":23,"memory_in_bytes":53502},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_28_2014-10-18_19-43-artworks":{"primaries":{"docs":{"count":6781,"deleted":0},"store":{"size_in_bytes":16296415,"throttle_time_in_millis":1465},"indexing":{"index_total":6781,"index_time_in_millis":18053,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":79900,"query_time_in_millis":210164,"query_current":0,"fetch_total":8701,"fetch_time_in_millis":1953,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":63,"total_time_in_millis":3943},"flush":{"total":5,"total_time_in_millis":557},"warmer":{"current":0,"total":68,"total_time_in_millis":198},"filter_cache":{"memory_size_in_bytes":421104,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":3568270,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":18,"memory_in_bytes":133950},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":6781,"deleted":0},"store":{"size_in_bytes":16296415,"throttle_time_in_millis":1465},"indexing":{"index_total":6781,"index_time_in_millis":18053,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":79900,"query_time_in_millis":210164,"query_current":0,"fetch_total":8701,"fetch_time_in_millis":1953,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":63,"total_time_in_millis":3943},"flush":{"total":5,"total_time_in_millis":557},"warmer":{"current":0,"total":68,"total_time_in_millis":198},"filter_cache":{"memory_size_in_bytes":421104,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":3568270,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":18,"memory_in_bytes":133950},"translog":{"operations":0,"size_in_bytes":0}}},"vm-stg_16_2014-10-29_17-36-artworks":{"primaries":{"docs":{"count":2348,"deleted":0},"store":{"size_in_bytes":10005645,"throttle_time_in_millis":0},"indexing":{"index_total":2348,"index_time_in_millis":20124,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":200,"time_in_millis":22,"exists_total":200,"exists_time_in_millis":22,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":13310,"query_time_in_millis":26575,"query_current":0,"fetch_total":4650,"fetch_time_in_millis":637,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":42,"total_time_in_millis":2679},"flush":{"total":5,"total_time_in_millis":226},"warmer":{"current":0,"total":47,"total_time_in_millis":82},"filter_cache":{"memory_size_in_bytes":9431,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":5924691,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":37,"memory_in_bytes":72047},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":2348,"deleted":0},"store":{"size_in_bytes":10005645,"throttle_time_in_millis":0},"indexing":{"index_total":2348,"index_time_in_millis":20124,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":200,"time_in_millis":22,"exists_total":200,"exists_time_in_millis":22,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":13310,"query_time_in_millis":26575,"query_current":0,"fetch_total":4650,"fetch_time_in_millis":637,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":42,"total_time_in_millis":2679},"flush":{"total":5,"total_time_in_millis":226},"warmer":{"current":0,"total":47,"total_time_in_millis":82},"filter_cache":{"memory_size_in_bytes":9431,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":5924691,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":37,"memory_in_bytes":72047},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_3_2014-10-18_20-30-artworks":{"primaries":{"docs":{"count":2154,"deleted":0},"store":{"size_in_bytes":4535425,"throttle_time_in_millis":0},"indexing":{"index_total":2154,"index_time_in_millis":16220,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":20510,"query_time_in_millis":809,"query_current":0,"fetch_total":511,"fetch_time_in_millis":59,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":26,"total_time_in_millis":3600},"flush":{"total":5,"total_time_in_millis":1003},"warmer":{"current":0,"total":31,"total_time_in_millis":12},"filter_cache":{"memory_size_in_bytes":688,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":183270,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":21,"memory_in_bytes":38252},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":2154,"deleted":0},"store":{"size_in_bytes":4535425,"throttle_time_in_millis":0},"indexing":{"index_total":2154,"index_time_in_millis":16220,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":20510,"query_time_in_millis":809,"query_current":0,"fetch_total":511,"fetch_time_in_millis":59,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":26,"total_time_in_millis":3600},"flush":{"total":5,"total_time_in_millis":1003},"warmer":{"current":0,"total":31,"total_time_in_millis":12},"filter_cache":{"memory_size_in_bytes":688,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":183270,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":21,"memory_in_bytes":38252},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_123_2014-10-16_19-04-authors":{"primaries":{"docs":{"count":5198,"deleted":0},"store":{"size_in_bytes":3418642,"throttle_time_in_millis":0},"indexing":{"index_total":5198,"index_time_in_millis":2487,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":38,"total_time_in_millis":546},"flush":{"total":5,"total_time_in_millis":174},"warmer":{"current":0,"total":43,"total_time_in_millis":29},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":33,"memory_in_bytes":63600},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":5198,"deleted":0},"store":{"size_in_bytes":3418642,"throttle_time_in_millis":0},"indexing":{"index_total":5198,"index_time_in_millis":2487,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":38,"total_time_in_millis":546},"flush":{"total":5,"total_time_in_millis":174},"warmer":{"current":0,"total":43,"total_time_in_millis":29},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":33,"memory_in_bytes":63600},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_123_2014-10-16_19-04-artworks":{"primaries":{"docs":{"count":25517,"deleted":0},"store":{"size_in_bytes":110101057,"throttle_time_in_millis":22613},"indexing":{"index_total":25517,"index_time_in_millis":97545,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":4,"time_in_millis":0,"exists_total":4,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":93780,"query_time_in_millis":1157526,"query_current":0,"fetch_total":16946,"fetch_time_in_millis":17213,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":265,"total_time_in_millis":18977},"flush":{"total":5,"total_time_in_millis":4079},"warmer":{"current":0,"total":257,"total_time_in_millis":434},"filter_cache":{"memory_size_in_bytes":2104703,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":8794383,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":37,"memory_in_bytes":624583},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":25517,"deleted":0},"store":{"size_in_bytes":110101057,"throttle_time_in_millis":22613},"indexing":{"index_total":25517,"index_time_in_millis":97545,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":4,"time_in_millis":0,"exists_total":4,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":93780,"query_time_in_millis":1157526,"query_current":0,"fetch_total":16946,"fetch_time_in_millis":17213,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":265,"total_time_in_millis":18977},"flush":{"total":5,"total_time_in_millis":4079},"warmer":{"current":0,"total":257,"total_time_in_millis":434},"filter_cache":{"memory_size_in_bytes":2104703,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":8794383,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":37,"memory_in_bytes":624583},"translog":{"operations":0,"size_in_bytes":0}}},"vm-dev_18_2014-10-18_18-45b-artworks":{"primaries":{"docs":{"count":10016,"deleted":0},"store":{"size_in_bytes":22417551,"throttle_time_in_millis":1484},"indexing":{"index_total":10016,"index_time_in_millis":16742,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":18787,"time_in_millis":1264,"exists_total":18787,"exists_time_in_millis":1264,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":324330,"query_time_in_millis":2485255,"query_current":0,"fetch_total":56533,"fetch_time_in_millis":7081,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":80,"total_time_in_millis":3436},"flush":{"total":5,"total_time_in_millis":188},"warmer":{"current":0,"total":86,"total_time_in_millis":315},"filter_cache":{"memory_size_in_bytes":758494,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":5433231,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":27,"memory_in_bytes":211797},"translog":{"operations":0,"size_in_bytes":0}},"total":{"docs":{"count":10016,"deleted":0},"store":{"size_in_bytes":22417551,"throttle_time_in_millis":1484},"indexing":{"index_total":10016,"index_time_in_millis":16742,"index_current":0,"delete_total":0,"delete_time_in_millis":0,"delete_current":0},"get":{"total":18787,"time_in_millis":1264,"exists_total":18787,"exists_time_in_millis":1264,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":324330,"query_time_in_millis":2485255,"query_current":0,"fetch_total":56533,"fetch_time_in_millis":7081,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":80,"total_time_in_millis":3436},"flush":{"total":5,"total_time_in_millis":188},"warmer":{"current":0,"total":86,"total_time_in_millis":315},"filter_cache":{"memory_size_in_bytes":758494,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":5433231,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":-1,"memory_size":"-1b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":27,"memory_in_bytes":211797},"translog":{"operations":0,"size_in_bytes":0}}}}}

Heureusement, des outils existent. Lorsque l’on est développeur, on utilise les bibliothèques de fonction Python ou Ruby voire Java. On fait son petit code et on le lance. Nous, nous sommes des admins, nous préférons la ligne de commande, les « pipes » (tubes) et les enchaînements. Nous allons voir comment jouer avec JSON lorsqu’on est un administrateur.

Continue reading Jouons avec JSON

Optimisation d’un script SHELL

 

Nous avons un script shell. Celui-ci fonctionne en deux étapes. La première étape dure une nuit. La deuxième étape dure elle aussi toute une nuit. Cependant, ce processus doit être exécuté pour 60 serveurs. La durée totale théorique est donc de 120 jours!

La première étape peut être parallélisée, sur les 60 serveurs en même temps. La première étape peut donc avoir une durée d’une nuit si on lance le processus sur les 60 serveurs en parallèle. La deuxième étape ne peut être parallélisée car elle doit être exécutée sur un seul et unique serveur et elle génère 60 résultats différents (un par serveur source, dépendant du résultat de l’étape 1 des serveurs sources). La 2e étape a donc une durée incompressible de 60 nuits.

Il faut donc au moins 61 nuits (1 + 60) pour exécuter l’intégralité du processus. Trop long. Trop complexe. Trop peu sûr. Comment optimiser cela?

Edit : précisions apportées suite au commentaire de « bob morane », merci à lui!

Continue reading Optimisation d’un script SHELL