Skip to main content

elasticsearch bulk add

Batch insert data into Elasticsearch
When I want to batch add documents to elasticsearch, this is how I do
I create an index:
curl -XPUT localhost:9200/tmp2
I prepare the file for batch insertion (official documentation)

{ "create" : { "_index" : "tmp2", "_id" : "1" } }
{"title": "RxJS: How to Use refCount","link": "https://blog.angularindepth.com/rxjs-how-to-use-refcount-73a0c6619a4e","text": "My previous article — Understanding the publish and share Operators — looked only briefly at the refCount method. Let’s look at it more closely here."}
{ "create" : { "_index" : "tmp2", "_id" : "12" } }
{"title": "I reverse-engineered Zones (zone.js) and here is what I’ve found","link": "https://blog.angularindepth.com/i-reverse-engineered-zones-zone-js-and-here-is-what-ive-found-1f48dc87659b","text": "Zones is a new mechanism that helps developers work with multiple logically-connected async operations. Zones work by associating each async operation with a zone."}
Save the file under the name "news.json"
I the perform the insertion:
curl -H "Content-Type: application/x-ndjson" -XPOST  localhost:9200/_bulk --data-binary "@news.json"
Pay attention to the fact that data must be one line: do not format you JSON data.
For example, this will lead to ERROR, because you put "\n" where Elasticsearch dont expect:

{ "create" : { "_index" : "tmp2", "_id" : "1" } }
{
 "title": "RxJS: How to Use refCount",
 "link": "https://blog.angularindepth.com/rxjs-how-to-use-refcount-73a0c6619a4e",
 "text": "My previous article — Understanding the publish and share Operators — looked only briefly at the refCount method. Let’s look at it more closely here."
}
{ "create" : { "_index" : "tmp2", "_id" : "12" } }
{
 "title": "I reverse-engineered Zones (zone.js) and here is what I’ve found",
 "link": "https://blog.angularindepth.com/i-reverse-engineered-zones-zone-js-and-here-is-what-ive-found-1f48dc87659b",
 "text": "Zones is a new mechanism that helps developers work with multiple logically-connected async operations. Zones work by associating each async operation with a zone."}
The ERROR will be:
{"error":
 {
  "root_cause":[
   {
    "type":"illegal_argument_exception",
    "reason":"Malformed action/metadata line [3], expected START_OBJECT but found [VALUE_STRING]"
   }],
  "type":"illegal_argument_exception","reason":"Malformed action/metadata line [3], expected START_OBJECT but found [VALUE_STRING]"
 },
  "status":400
}


Popular posts from this blog

npm run build base-href

Using NPM to specify base-href When building an Angular application, people usually use "ng" and pass arguments to that invocation. Typically, when wanting to hard code "base-href" in "index.html", one will issue: ng build --base-href='https://ngx.rktmb.org/foo' I used to build my angular apps through Bamboo or Jenkins and they have a "npm" plugin. I got the habit to build the application with "npm run build" before deploying it. But the development team once asked me to set the "--base-href='https://ngx.rktmb.org/foo'" parameter. npm run build --base-href='https://ngx.rktmb.org/foo did not set the base href in indext.html After looking for a while, I found https://github.com/angular/angular-cli/issues/13560 where it says: You need to use −− to pass arguments to npm scripts. This did the job! The command to issue is then: npm run build -- --base-href='https://ngx.rktmb.org/foo...
Read more

wget maven ntlm proxy

How to make wget, curl and Maven download behind an NTLM Proxy Working on CentOS, behind an NTLM proxy: yum can deal without problem with a NTLM Proxy wget, curl and Maven cannot The solution is to use " cntlm ". " cntlm " is a NTLM client for proxies requiring NTLM authentication. How it works Install "cntlm" Configure "cntlm"  by giving it your credentials by giving it the NTLM Proxy Start "cntlm" deamon (it listens to "127.0.0.1:3128") Configure wget, curl and Maven to use "cntlm" instead of using directly the NTLM Proxy Note: You will have then a kind of 2 stages Proxy : cntlm + the NTLM proxy Configure CNTLM After installing cntlm, the configuration file is in "cntlm.conf". You must have your domain (in the Windows meaning), proxy login and  proxy password. Mine are respectively: rktmb.org, mihamina, 1234abcd (yes, just for the example) You must have you NTLM Proxy Hostnama or IP ...
Read more

Undefined global vim

Defining vim as global outside of Neovim When developing plugins for Neovim, particularly in Lua, developers often encounter the "Undefined global vim" warning. This warning can be a nuisance and disrupt the development workflow. However, there is a straightforward solution to this problem by configuring the Lua Language Server Protocol (LSP) to recognize 'vim' as a global variable. Getting "Undefined global vim" warning when developing Neovim plugin While developing Neovim plugins using Lua, the Lua language server might not recognize the 'vim' namespace by default. This leads to warnings about 'vim' being an undefined global variable. These warnings are not just annoying but can also clutter the development environment with unnecessary alerts, potentially hiding other important warnings or errors. Defining vim as global in Lua LSP configuration to get rid of the warning To resolve the "Undefined global vi...
Read more