Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
"Anything that can go wrong will go wrong" - Murphy's Law
Looks like someone hasn't follows the installation instructions correctly...
node-gyp (used to build better-sqlite3) is not compatible with Python 3. Yes, it's been 10 years since 3's release. Yes, it sucks. But you need to have python 2.7.x in your path to install better-sqlite3. If you're not using python anywhere else, uninstall it and make sure to follow my installation instructions. If you are, run npm config set python python2.7
before running the install again.
When using persistent enmaps, it's very important to understand that it takes time to load the data from the database. Attempting to use the enmap before it's fully loaded can lead to errors, so we need to make sure it's ready before using it.
To make sure that all your data is loaded before you start working, Enmap provides a handy property called defer
, which is a promise that is resolved once the provider is ready and all the data has been loaded into memory. There are a few ways to use defer
, since it's a promise.
For more information on async/await and promises, see My JavaScript Guide.
Enmap also provides a isReady
option that tells you if the database is loaded. You can use that however you want, though the preferred method is using defer
.
In order to install Enmap, you'll need a few things installed on your machine. First off, you need NodeJS (Node 10 is required. Not 8, not 9, not 12. NODE 10 ONLY). For Windows and MacOS, simply download and install from the website. For Linux, see this page for installation.
To install Enmap in your project, all you need to do is run the following command:
This may take a few minutes, then you're ready to use it.
For persistence you need to also install better-sqlite-pool
, which is necessary for the sqlite database interaction.
better-sqlite-pool
has a specific pre-requisite which is needed to build it. How to install these depends on your operating system, so see below for instructions:
On Windows, two things are required to install enmap-sqlite. Python 2.7 and the Visual Studio C++ Build Tools. They are required for any module that is built on the system, which includes sqlite.
The Windows Build Tools require over 3GB of space to install and use. Make sure you have enough space before starting this download and install!
To install the necessary pre-requisites on Windows, the easiest is to simply run the following command, under an administrative command prompt or powershell:
It's very important that this be run in the administrative prompt, and not a regular one.
Once the windows-build-tools are installed (this might take quite some time, depending on your internet connection), close all open command prompts, powershell windows, and editors with a built-in console/prompt. Otherwise, the next command will not work.
On Linux, the pre-requisites are much simpler in a way. A lot of modern systems (such as Ubuntu, since 16.04) already come with python 2.7 pre-installed. For some other systems, you might have to fiddle with it to either get python 2.7 installed, or to install both 2.7 and 3.x simultaneously. Google will be your friend.
As for the C++ build tools, that's installed using the simple command: sudo apt-get install build-essential
for most debian-based systems. For others, look towards your package manager and specificall "GCC build tools". Your mileage may vary but hey, you're using Linux, you should know this stuff.
As of writing this page, MacOS versions seem to all come pre-built with Python 2.7 on the system. You will, however, need the C++ build tools.
Install XCode
Once XCode is installed, go to Preferences, Downloads, and install the Command Line Tools.
Once installed, you're ready to continue.
Once those pre-requisites are installed, simply run the following command:
This will take a few minutes also, as it needs to build the module from source code.
This guide assists in migrating your data from Enmap 3 using Providers, to the latest version of enmap.
You do not need this page if you're new to Enmap or if you're starting a new project!
Upgrading to enmap v4 requires a little bit of migration, as Enmap 4 changed the internal method by which data is stored, slightly. To use this migration:
Make a copy of your current app in a new folder.
Create a new folder "on the same level" as your bot. Name it something like "migrate"
You should now have 3 folders. Something like mybots/coolbot
, mybots/coolbot-copy
, mybots/migrate/
In the migrate
folder, run npm i enmap@3.1.4 enmap-sqlite@latest
, as well as whatever source provider you need if it's not sqlite (in my example, npm i enmap-mongo@latest
You should now have something like the following image.
In the migrate
folder, create an index.js
and use the following script for migration. Note that it's an example, change the provider option to fit what you're actually using.
Very important: the "target" must be enmap-sqlite. Enmap v4 only supports an sqlite-backend.
From the migrate
folder, run node index.js
, which should correctly migrate your data.
If you're using enmap-sqlite already, you don't really need to do the entire thing above. Adding a single file called migrate.js
to your project folder, then running it with node migrate.js
will convert the format and then all you need is to modify the code for Enmap 4. Stilll, I recommend backing up your bot first. Just in case.
There is very little you need to change when moving to Enmap 4. The only changes that are required after migrating is the initialization of your Enmap which is now simpler.
If using Enmap.multi(), the change is just as simple:
The rest of your code (all interactions with Enmap) can remain the same - there should be no need to edit any of it.
Once your data is migrating and the code is changed, you can go ahead and install enmap version 4 through npm i enmap@latest
in your "new" bot folder (the target of the migration). This will take a few minutes (it needs to rebuild sqlite) and output that 4.0.x is now installed. Start the bot, and it should be working! If it doesn't, join the support server and we'll help you out ^_^.
Enmap stands for "Enhanced Map", and is a data structure based on the native JavaScript Map() structure with additional helper methods from the native Array() structure.
LA DOCUMENTATION EST PRÉSENTEMENT EN TRADUCTION
Enmap stands for "Enhanced Map", and is a data structure based on the native JavaScript Map() structure with additional helper methods from the native Array() structure.
Enmap also offers persistence, which means it will automatically save everything to save to it in a database, in the background, without any additional code or delays.
Enmap requires filesystem access. It DOES NOT WORK on Heroku, or other such systems that do not allow you to save data directly to disk.
While there are other better-known systems that offer some features of Enmap, especially caching in memory, Enmap is targetted specifically to newer users of JavaScript that might not want to deal with complicated systems like Redis for caching, or database queries.
Here are some advantages of using Enmap:
Simple to Use: Basic enmap usage can be completely done with 1-2 lines of initalization, and 3 commands, set(), get() and delete().
Very Fast: Since Enmap resides in memory, accessing its data is blazing fast (as fast as Map() is). Even with persistence, Enmap still only accesses data from memory so you get it almost instantly.
Some disadvantages, compared to using a database connection directly:
More memory use: Since Enmap resides in memory and (by default) all its data is loaded when it starts, your entire data resides in RAM. When using a large amount of data on a low-end computer or VPS, this might be an issue for some users.
Limited power: You can have multiple Enmap "tables" loaded in your app, but they do not and cannot have relationships between them. Basically, one enmap value can't refer to another value in another enmap. This is something databases can be very good at, but due to the simplistic nature of Enmap, it's not possible here.
Lack of scalability: Enmap is great for small apps that require a simple key/value storage. However, a scalable app spread over multiple processes, shards, or clusters, will be severely limited by Enmap as it cannot update itself from the database on change - one process would not be aware of another process' changes.
Mostly, this documentation will be concentrating on the "persistent" version of enmap - the one where data is saved automatically.
If you don't want persistence, the only difference is how you initialize the enmap:
Using persistent enmaps require the additional install of the better-sqlite-pool
module.
If using a persistent enmap, you need to add options:
The following is a list of all options that are available in Enmap, when initializing it:
name
: A name for the enmap. Defines the table name in SQLite (the name is "cleansed" before use).
If an enmap has a name, it is considered persistent and will require better-sqlite-pool
to run.
If an enmap does not have a name, it is not persistent and any option related to database interaction is ignored (fetchAll, autoFetch, polling and pollingInterval).
fetchAll
: Defaults to true
, which means fetching all keys on load. Setting it to false
means that no keys are fetched, so it loads faster and uses less memory.
autoFetch
: Defaults to true
. When enabled, will automatically fetch any key that's requested using get, getProp, etc. This is a "syncroneous" operation, which means it doesn't need any of this promise or callback use.
dataDir
: Defaults to ./data
. Determines where the sqlite files will be stored. Can be relative (to your project root) or absolute on the disk. Windows users , remember to escape your backslashes!
cloneLevel
: Defaults to deep
. Determines how objects and arrays are treated when inserting and retrieving from the database.
none
: Data is inserted by reference, meaning if you change it in the Enmap it changes outside, and vice versa. This should only be used in non-persistent enmaps if you know what you're doing!.
shallow
: Any object or array will be inserted as a shallow copy, meaning the first level is copied but sub-elements are inserted as references. This emulates Enmap 3's behaviour, but is not recommended unless you know what you're doing.
deep
: Any object or array will be inserted and retrieved as a deep copy, meaning it is a completely different object. Since there is no chance of ever creating side-effects from modifying object, This is the recommended, and default, setting.
NEW IN ENMAP 4.2.0 AND HIGHER
polling
: defaults to false
. Determines whether Enmap will attempt to retrieve changes from the database on a regular interval. This means that if another Enmap in another process modifies a value, this change will be reflected in ALL enmaps using the polling feature.
pollingInterval
: defaults to 1000
, polling every second. Delay in milliseconds to poll new data from the database. The shorter the interval, the more CPU is used, so it's best not to lower this. Polling takes about 350-500ms if no data is found, and time will grow with more changes fetched. In my tests, 15 rows took a little more than 1 second, every second.
This page is a work in progress and may not have the polish of a usual Evie-Written document!
Some quick docs:
Possible Operators (accepts all variations listed below, as strings):
+
, add
, addition
: Increments the value in the enmap by the provided value.
-
, sub
, subtract
: Decrements the value in the enmap by the provided value.
*
, mult
, multiply
: Multiply the value in the enmap by the provided value.
/
, div
, divide
: Divide the value in the enmap by the provided value.
%
, mod
, modulo
: Gets the modulo of the value in the enmap by the provided value.
^
, exp
, exponential
: Raises the value in the enmap by the power of the provided value.
Simple to Install: Enmap itself only requires a simple npm install
command to install and use, and a single line to initialize. When using persistent providers, some additional pre-requisites are necessary. .
While Enmap is a Map enhanced with Array methods, Enmap also offers some enhanced array methods for the data stored inside of it. Talk about ArrayCeption!
So what do I mean by methods for your stored data? I mean that you can store arrays inside Enmap, and directly push, pull, add and remove from those arrays. There are methods to work both on direct arrays, as well as arrays stored inside of an object.
Let's take a look at two example entries in enmap that we can use. The first is a direct array, the second is an array inside an object.
There are two methods to push to an array, one for simple arrays and one for arrays inside objects. Pushing in an enmap array is the same as a regular array push: it adds the element to the end of the array.
The second parameter in pushIn is the "path" to the array in an object. It works the same as the properties path used in Working With Objects.
Similarly, you can remove from an array. Note, however, that this will only work if you're removing a string or number. Removing an object from an array is not supported.
Enmap is a great way to store structured data, and offers a few helper features that directly affect both objects and arrays.
Let's assume for a moment that we want to store the following data structure in Enmap:
This structure has 5 "properties": first
, second
, changeme
, isCool
, sub
. The sub
property has 2 properties of its own, yay
and thing
.
To store this structure in Enmap, you can use a variable, or just straight-up write the object:
Note: All further methods require the value to be an object. If you attempt to get, set, modify or remove using the below methods and your value isn't an object, Enmap will throw an error.
Retrieving a specific property from an object is done through the get()
method, by specifying both the key and the "path" to the property you want.
The exact method is <Enmap>.get(key, path)
.
You can also check if a specific property exists or not. This is done through the has
method, with a key, and path to the property:
There are a few various ways to modify properties of both Objects and Arrays. The very basic way to set a property on an object or array is through .set(key, value, path)
like the following examples:
As you can see, setProp() and getProp() work on the same concept that the path can be as complex as you want.
Arrays have additional helper methods, you can see them here.
Now that we have a functional Enmap structure (which we'll always refer to as myEnmap
), we're ready to start writing data to it, and getting data from it.
The code samples on this page assume that you have correctly initialized
myEnmap
, and awaited its initialization if it's persistent.
In terms of Enmap, "writing", "adding" and "editing" data is essentially the same thing. When using the basic set()
method, if the key does not exist it's created, and if it does, it's modified.
Enmap supports most native JavaScript data types, with a few small exceptions.
null
and undefined
values are not supported.
Complex objects like Set()
, Map()
, etc, are not supported.
Class instances and Functions are not supported.
As a general rule, except for null and undefined, anything that can be handled by JSON.stringify()
will work as expected in Enmap. This includes:
String
Number
Integer
Boolean
Object
Array
Objects and Arrays are a little more complex to deal with, so they have their own page. See Working with Objects for more information.
The usage for the set()
method is simple:
key
must be a string or integer. A key should be unique, otherwise it will be overwritten by new values using the same key.
value
must be a supported native data type as mentioned above.
Here are a few examples of writing simple data values:
Getting data back from an Enmap is just as simple as writing to it. All you need is the key
of what you want to retrieve, and you get its value back!
That's pretty much it for only retrieiving a single data value. There are more complex operations that are available, take a look at Array Methods for the more advanced things you can do on Enmap's data!
For this purpose, there are features in Enmap that enable less caching, by sacrificing some speed and ease of use. That is to say, with data not being fully loaded, there are some things that can't be done easily - see below for details.
The options are as follow:
fetchAll
: Defaults to true
, which means fetching all keys on load. Setting it to false
means that no keys are fetched, so it loads faster and uses less memory.
autoFetch
: Defaults to true
. When enabled, will automatically fetch any key that's requested using get, getProp, etc. This is a "syncroneous" operation, which means it doesn't need any of this promise or callback use.
If fetchAll is set to false, no data (by default) will be loaded from the database - the Enmap will be completely empty. This means that doing a .size
check returns 0, looping and filtering doesn't return anything, and get() requests all return null.
Ok but... how's that useful? It's useful because if you don't need the data, it's not loaded. To load data, there are 2 different methods available.
enmap.fetchEverything()
will, of course, fetch all the data from the database - this is the method that's called on init() if fetchAll is true. This means you can stagger your loading time, or that you can decide, later, under certain conditions, you do want to load everything.
enmap.fetch(key)
can fetch a single key from the database, saves it to enmap, and returns a promise with the fetched value.
enmap.fetch([array, of, keys])
will fetch each key in the requested array, and return an array of [key, value]
pairs for each fetched value.
enmap.count
will give you the number of keys in the database itself, including uncached ones (.size is only cached values).
enmap.indexes
will give you a list of keys in the database, including uncached ones.
Yup. Those are the only things you really need to know for the current version of Enmap's fetchAll feature.
I'm working on the following features in future versions of enmap, related to fetch methods:
Add the ability to check if the DB has a key without fetching (a sort of "uncached has()")
Add an auto-uncache feature so that "stale" keys are cleaned. This would combine well with autoFetch in that it would ultimately keep memory usage low.
This page will describe how to use Enmap from multiple files within your same project. Note that I mean the same app, process, or shard, but different files within this one running process.
When Enmap is used with its default options, it loads everything in its cache and generally provides your data from this cache, not directly from the database. In the case where you want to use the data from one Enmap from multiple locations, you might encounter the following issue:
Hi! When I update data in Enmap from one file, it doesn't update in the other file, I have to restart the bot to update. Is this a bug?
However, this also means that when you do new Enmap({ name: "something" })
from more than one file, that's also a different instance, that doesn't share the same memory space. So not only will it not update the data in memory for the other file, it also uses double the memory. And of course, that's bad. So how do we fix this?
Admitedly, the vast majority of you Enmap users are doing Discord.js Bots, and even though Enmap works fine with any nodejs project that need simple data storage, bots are my main clients. Considering this fact, we have an extremely simple way to share an Enmap between multiple files: We attach it to the bot client. Usually your client is defined in your main file (index.js, app.js, bot.js, whatever you named it), and every part of your bot has access to this client. We can attach Enmap directly to it, like so:
This will work even if you're using a command handler, framework, or whatever - as long as you have access to a client variable, you have access to your enmaps.
In other frameworks and libraries, you might have something similar. For example with Express or Koa for http servers, you can sometimes attach the enmap to your request from the very top, in a middleware. If that's not possible, or if you find that to be complicated, you can use the next method.
This means you can simply require that file elsewhere. Let's say we called that file db.js
, here's how you'd use it:
And as I mentioned, as a bonus you now have the ability to create functions which you can export and use, to simplify your code and remove duplication. So, let's say I need to get all the tags for a specific guild, and my tags are built using an object as shown above. To get all those tags for a guild, you'd need filters, right? Like so:
now let's say you use this code a lot in your app, and you'd like to not have to type this whole thing every time. You could add a simple function in your module that only takes an ID and returns the tags:
As described in , one disadvantage of Enmap is that it loads all your data in memory, so you're sacrificing RAM in order to gain speed. In larger projects, this might become a concern fairly quickly - or when using larger data sets that take more memory.
To answer my own obvious question: it's not a bug, it's a feature that I cannot implement. The way Enmap's cache works is that the data is loaded in memory in that of Enmap, and only for that instance. This is what enables you to have many different Enmaps in your project - one Enmap doesn't share data with another.
Important Note: Do NOT override Discord.js' existing collections! That means, client.users, client.guilds, etc. - none of these should be overriden.
All things considered, are probably the recommended way to use your Enmap in multiple files within your project. Not only does it give you a single file to import, lets you define multiple Enmaps you can individually import, it also gives you the ability to add specific functions to do common actions you use throughout your project.
As covered in , modules are fairly straightforward. This is how I have done an Enmap shared module before:
And there you have it! There are other ways to build the exports, you can also split it differently, take a look at for more information.
In this example we'll be using Enmap to store user data in order to authenticate users on a simple Koa application. In order to make this secure, we'll be using bcrypt
to encrypt the passwords, so of course they will not be plain text in the database.
We'll be using koa
as a web server module, along with koa-session
in order to store the session information. Additionally, for ease of use, koa-router
is used to simplify the route code.
This tutorial uses
koa-session
which, by default, is insecure since it stores the entire session data in a browser cookie. This means the password, though encrypted, would be availble in the cookie, and easy to spoof. There are many session stores available for different storage system, but using them is beyond the scope of this example.
To install those requirements, run the following in a new, empty folder for your project:
Once all of those are installed, we're ready to start! We're going to create an index page, login page, and one "protected" page that requires login. Let's start with the top of the file, which is all the required modules:
So now we have all the basics necessary.
Let's create a few functions specifically for the login features, related to enmap. First, a function to create a new user:
This function takes in the following arguments:
username
which obviously is self-explanatory: it's the username entered during login.
name
is the "full name" or "display name", for a friendly display on the page or an email.
plainpw
is the password desired for this account. It has to come in as plain text, obviously, in order to be properly saved in the database.
admin
is a boolean value representing whether the user should be administrator. If creating something like a blog, only an administrator could create other administrators. It's false by default.
The function first checks if the username exists and returns an error if it does. It then generates a salted, hashed version of the password which it stores in the database. Don't let the name fool you, the password is not "encrypted", which implies that it can be decrypted. Instead, it's a "cryptographic hash functions", a unidirectional function that cannot be undone. The only way to verify that a password is correct is to re-hash it again and compared the hashes.
Once the hashed password is obtained, the user itself is stored in the database with all 4 incoming arguments except the password which is the hashed version.
The login function takes in the username and the incoming plain password and verifies that the hashed version corresponds with the one stored in the database.
An important point here is that this function returns a promise in all cases. If the username doesn't exist or the password is blank, a false
response is returned in a promise. Otherwise, the response of bcrypt's compare
function is returned. This function returns true if the passwords match, false if they do not.
There's a few configuration items we need to take care of. First off, the session settings:
Then we need to setup how Koa will handle rendering EJS pages. This is one pretty awesome thing about Koa, that this can be setup automatically and globally, but don't let me gush all over this!
So let's establish our "routes", which is the pages that can be accessed by the browser. With the help of the Router, this can be really straightforward.
Then we have the login route, which does a lot of the bulk of our work. It checks for login, and adds everything it needs to the session in Koa if the authentication is successful:
Let's also create a logout function, that simply destroys the current session and returns the user to the index:
This one is pretty straightforward, so I don't think I need to get into the details, right? ;)
Lastly, we have the route for our "private" page. the one that only works if you're logged in. Now, there are "better" ways to establish protected routes, but let's go with the simplest one for now. We're just going to check for the logged
property of the session to determine if the user is logged in.
At the very end of our file we still have a bit of stuff to add. Mainly starting the server, but also telling the parser and routers to initialize. This would be how it's done:
While templating is slightly beyond the scope of an authentication tutorial, I would be remiss to ignore the fact that logging in without a page would be... let's say a little hard.
Koa's EJS templating configuration, that we did above, means that templates need to appear in the views
folder. There will be a few template files:
template.html
will be the "main" template. It will have the header, footer, and whatever else we want to appear on every page.
index.html
will be the main page everyone can access.
login.html
contains the login page and form
secret.html
has a little secret about something.
Let's start with the template.html
file.
The <%- body %>
tag is where the contents of the other pages appear.
The index.html
is just some welcome thingy that we aren't concerned about:
Then we have the login.html
page which has our form. The form simply posts back to itself, so it'll trigger the .post
endpoint:
And, finally, the secret.html
page we've all been waiting for. Nothing that you haven't heard before, though:
With all said and done, This should be your project. Want a more "advanced" version of this project? Check out My full blogging platform, Koarrots.
This points bot is simple, but functional. Make sure you've followed the Installation Instructions before doing the following.
First, you need to create a new persistent Enmap. Here's how it goes:
That will create a new Enmap under the name of points, and attaches it to the client object so it can be used where ever you have access to the client object.
The obvious goal of a points system is to accumulate fake internet points and gloat about it. So, of course, that's going to be our first focus. In this example implementation, we will make the points guild-specific, and unuseable in DMs. Points will still accumulate even if the user does a command, which simplifies our code a bit.
Our starting point is a very basic message handler with pre-existing commands - such as what we see in the Command with Arguments page on An Idiot's Guide. The code is as such:
We do have a small caveat - we really don't want to react on Direct Messages, so our whole code will be in a block that checks for that:
Our very first step is going to be to initialize a new entry in the enmap for any new user - one we haven't received a message from before. This is done using the enmap.ensure(key, defaultvalue)
method, which can check if a specific key exists in the Enmap, write it if it doesn't (and return the defaultvalue in this case). Note that our keys take the format of guildid-userid
so they're unique to the guild and the user. Also, our data in this case is a complete object, which we'll take advantage of fully.
There's obviously a few ways we could have done this, including some fancy ternary condition or whatever. I will, however, keep this code as simple to read as possible.
The following bit is super simple - Enmap has a method to directly increment a value in Enmap even if it's in an object. Pretty clever if I do say so myself!
Have your own way of incrementing points? No problem! Enmap.math() gives you the ability to add, multiply, and act upon any numerical value or property. To add 10 points, for instance, client.points.math(key, "+", 10, "points")
would be used.
Time to level up! If a user has enough points, they will go up a level. Now we have to do some math here, but don't run off in fear, this one's pretty easy. This is how we calculate the levels:
This line will calculate the square root of currentPoints
then multiplies that result by 0.1 then floors that result for a round number.
Now we should work out if you've amassed enough points to actually level up, by grabbing the current user's level and comparing them. If the new calculated level is higher, it means the user leveled up and we can deal with that, first by sending them a very annoying mee6-inspired message!
Lastly, we want to update the score.level
value with the new level so throw this under the message.reply
.
So here's the whole thing from top to bottom, with bonus comments!
Alright, that's the bulk of the code, you could throw this into your bot and it would work like a charm, however your users wouldn't know how many points, or even their levels, so let's fix that, make a new command called points
, which will also show them their level.
Obviously there's no way for us to know how you're making commands, so again we'll assume you're doing a bot in a single js file. You may need to adjust the code, of course!
So let's re-iterate our current starting position.
The points
command would look like this:
Let's finish this off with a very simple leaderboard
command that will show the top 10 users in the current guild. For this we'll need to filter the Enmap to only get the users for the current guild, then we'll convert the results to an array, sort that, and keep the first 10 results only.
We convert to an array because an Enmap
, just like its underlying Map
structure, is not ordered and thus cannot be sorted. It may seem ordered because it stores by keys, but that's actually a quirk, not a feature.
So here's our leaderboard command:
Removed all Provider support. Enmap is now locked to an sqlite
database. This is because having a single provider is much easier to maintain, and gives me possibilities for features that were not previously available. The entire autoFetch/fetchAll system is possible only through the use of better-sqlite3
, which is sync but non-blocking. Existing providers will remain valid for enmap 3, and updates could still happen to enmap 3 and providers in the future, especially bug fixing. See Upgrading for more details.
New option: autoFetch
: Automatically fetches uncached keys when getting data from enmap. See Using the fetchAll option for more details.
Better error descriptions: All methods now verify the database is ready before running. This prevents confusion if trying to get or set data before the database is loaded.
New method: count
: This method returns the total number of keys for the enmap, even if they aren't cached. More Details
New method: indexes
: This method returns an array of keys for the enmap, even if they aren't cached. Similar to enmap.keyArray()
, but useful when fetchAll is false. More Details
New method: evict
: This method takes a key name or array of key names, and removes them from the cache. It does not delete the data, only "uncaches" it from Enmap. Useful when fetchAll is false, and you want to reduce memory usage. More Details.
New method: ensure
: This method can be used to ensure that a key exists in the enmap, and create it if it doesn't. It's a shortcut to the "if !has(key) set(key, value) get(key)" pattern in code. More Details.
From the first moment where I started using the Discord.js library, one thing in it fascinated me: "Collections". Discord.js Collections are a Map structure from JavaScript on top of which a bunch of useful methods are added, most notably those from JavaScript's Array structure.
Things like map, filter, reduce, find, sort... they made Maps so useful, so much more powerful, that I admired their design. It struck me at one point, that if such a structure were to be separated from Discord.js and perhaps made to be saved in a database, it would make interacting with data so easy that even a child could do it.
So when I started getting seriously into bot that required their own data to be saved, I turned to Amish Shah (Hydrabolt) and I said to him, I said "Listen, buddy, can I extract Collections and publish them as a separate module? That'd be awesome!" and Amish replied, like the great guy he is, "uhhhh sure, why not?"
And so, in May 2017, the djs-collection
module was born. It was a simple thing, just straight-up lifted from Discord.js' code (not illegally, mind you, I retained all proper licenses and credits to Hydrabolt!). The following month, I added persistence (saving to a database) to it and published djs-collection-persistent
, which then became my own defacto way to save data to a database.
But... let's be honest, npm install --save djs-collection-persistent
is a mouthful to type out. Plus, I was realizing that having those two as separate modules meant I had to update them separately and ensure they still worked individually... So at one point, I decided it was time to merge them.
Releasing a single repository meant that I could now change the name, because of the aformentioned "omg mile-long name" problem, and just the overall annoyance of writing it. So I settled on "well, they're enhanced maps, let's call it Enmap!". A quick search revealed Enmap's only other apparent meaning was that it was the name of a satellite, and I was guessing no one would confuse the two.
But I didn't want to force enmap users to have persistence, so at the same time I actually created a separate module called enmap-level
, which controlled the database layer and was completely optional. These modules I called Providers since, obviously, they provided data persistence and and API.
Enmap 0.4.0 was released at the beginning of October 2017, and since then has grown into a fairly solid module used by tens of thousands of people across the world, not only in discord.js bots but also in other projects built with Node. Its solidity and simplicity makes it the ideal storage for simple key/value pairs, and I'm extremely proud to have made it.
At the moment of writing this (2018-09-02) Enmap has been downloaded over 32,000 times and is growing by the minute with almost 10,000 downloads in August alone!
It's been a year now since I last wrote this post. And in case you were wondering, growth hasn't stopped! In fact, it's quite accelerated. One big change that I made was to go back to a single locked-in database provider, which I describe in Why SQLite Only?
Other than that, and adding some new features due to the switch to better-sqlite3, the main event is that just las month I reached a whopping 500,00 downloads for Enmap. Yes, that's a half million downloads for my little useful module that I started making for myself and ended up being useful for so many.
So one of the major changes from Enmap 3 to 4 is the removal of Providers. Providers were something I've had since Enmap 1.0 (when I converted from djs-collections-persistent), and had 2 advantages (2 reasons to have them in the first place).
It enabled supporting more databases, not only one. This gave more power to users, and, I thought, more capabilities.
It separated the memory enmap (non-persistent) from the database layer, so installing enmap didn't require installing sqlite.
But, after a year of updating Enmap, I realized that I'd painted myself in a corner with Providers. There came to light that there were multiple limitations to providers:
Features were limited to the "lowest common denominator", whatever was available to all providers. For instance, better-sqlite3 is a synchronous module that's nonblocking (which is a magical thing, really). But since all other providers required promises, then I had to use sqlite as a promise module.
Maintaining multiple providers is hard work. Every new feature would require updating all the providers (5 at this time), and there were many requests to create new providers which is an annoying, sometimes complicated task that adds even more work in the future.
There were features I wanted that simply weren't possible, physically, with the providers (like the fetchAll/autoFetch options).
In addition, the advantages became lesser with time. I realized most people were using leveldb at first, then most switch to sqlite when I updated guides to use that provider. Essentially, most people use whatever they're told to use. So, just forcing one database wasn't that much of an issue and didn't affect the majority of users.
Also, most people did use enmap with persistence, and those that didn't... well, most users have enmap to use with discord.js bots in the first place which gives them Collection - almost the same as a non-persistent enmap.
The reasoning behind removing all other providers and keeping sqlite was for specific features and capabilities inherent to the module I'm using, better-sqlite3.
better-sqlite3 is, as I mention, synchronous , which means, no callbacks, no promises. Just straight-up "make a request and it does it before the next line". No more need for "waiting" for things, resolving promises, etc.
The sync nature of better-sqlite3 means I can add an autoFetch feature. I can simply say "If the key isn't there, try to get the data", without requiring the user to resolve a promise. This is awesome.
By the same token, I can also add simple things like "get all keys in the database" using a getter. This means you can do enmap.indexes
and this is actually querying the database seamlessly without the user really knowing it does that. Same for enmap.count
and other features I'm planning.
So overall, I'm happy with my decision. It gives me more power, it gives users more features, and the people affected by the removal of the other providers are few and far between. Hell, enmap-pgsql has less than 1000 downloads on npm which is mostly mirrors and caches. It showed me that provider was pretty useless in the first place.
I recognize that some people might want to use enmap and can't use sqlite. This is for many valid reasons, for example using it on heroku which doesn't support sqlite and leveldb. For those users, I'm keeping the providers open for maintenance. If someone wants to maintain and update the V3 branch, or even fork the entire system and maintain it under a new name, I have no issue with that (assuming licenses are properly kept). I'll accept PRs on all enmap repositories, including backporting some features and adding new ones.
I'm also keeping the V3 docs in this gitbook so it can be maintained through gitbook and PRed on github.
You can still install any provider as you would before, and install enmap using npm i eslachance/enmap#v3
for the version 3 branch that will remain.
Want to help out? Join the Discord, fork and PR the Github for enmap, contribute!
This example uses a very, very simple bot made in discord.js to demonstrate how easily Enmap can be used to create a per-server configuration system.
Remember to follow the Installation Instructions before running any of this!
Our first task is of course to initialize the enmap correctly. In this case, we're attaching the settings to our client object so we can use it in different commands.
The main event for our bot, where messages are received. Any error here will probably crash the bot on every message received, so be careful!
The full, boring, unadultered enmap docs.
Kind: global class
Extends: Map
Enmap ⇐ Map
instance
.count ⇒ integer
.indexes ⇒ array.
.autonum ⇒ number
.get(key, path) ⇒ *
.fetch(keyOrKeys) ⇒ Enmap
| *
.close() ⇒ Promise.
.has(key, path) ⇒ boolean
.hasProp(key, path) ⇒ boolean
.destroy() ⇒ null
.array() ⇒ Array
.keyArray() ⇒ Array.
.random([count]) ⇒ *
| Array.
.randomKey([count]) ⇒ *
| Array.
.findAll(prop, value) ⇒ Array
.findKey(propOrFn, [value]) ⇒ string
| number
.exists(prop, value) ⇒ boolean
.sweep(fn, [thisArg]) ⇒ number
.filterArray(fn, [thisArg]) ⇒ Array
.partition(fn, [thisArg]) ⇒ Array.
.map(fn, [thisArg]) ⇒ Array
.some(fn, [thisArg]) ⇒ boolean
.every(fn, [thisArg]) ⇒ boolean
.equals(enmap) ⇒ boolean
static
.multi(names, options) ⇒ Array.
Initializes a new Enmap, with options.
Param
Type
Description
iterable
iterable
| string
If iterable data, only valid in non-persistent enmaps. If this parameter is a string, it is assumed to be the enmap's name, which is a shorthand for adding a name in the options and making the enmap persistent.
options
Object
options.name
string
The name of the enmap. Represents its table name in sqlite. If present, the enmap is persistent. If no name is given, the enmap is memory-only and is not saved in the database. As a shorthand, you may use a string for the name instead of the options (see example).
options.fetchAll
boolean
Defaults to true
. When enabled, will automatically fetch any key that's requested using get, getProp, etc. This is a "syncroneous" operation, which means it doesn't need any of this promise or callback use.
options.dataDir
string
Defaults to ./data
. Determines where the sqlite files will be stored. Can be relative (to your project root) or absolute on the disk. Windows users , remember to escape your backslashes!
options.cloneLevel
string
options.polling
boolean
defaults to false
. Determines whether Enmap will attempt to retrieve changes from the database on a regular interval. This means that if another Enmap in another process modifies a value, this change will be reflected in ALL enmaps using the polling feature.
options.pollingInterval
string
defaults to 1000
, polling every second. Delay in milliseconds to poll new data from the database. The shorter the interval, the more CPU is used, so it's best not to lower this. Polling takes about 350-500ms if no data is found, and time will grow with more changes fetched. In my tests, 15 rows took a little more than 1 second, every second.
options.ensureProps
boolean
defaults to false
. If enabled and the value in the enmap is an object, using ensure() will also ensure that every property present in the default object will be added to the value, if it's absent. See ensure API reference for more information.
options.strictType
boolean
defaults to false
. If enabled, locks the enmap to the type of the first value written to it (such as Number or String or Object). Do not enable this option if your enmap contains different types of value or the enmap will fail to load.
options.typeLock
string
Only used if strictType is enabled. Defines an initial type for every value entered in the enmap. If no value is provided, the first value written to enmap will determine its typeLock. Must be a valid JS Primitive name, such as String, Number, Object, Array.
integer
Retrieves the number of rows in the database for this enmap, even if they aren't fetched.
Kind: instance property of Enmap
Returns: integer
- The number of rows in the database.
array.
Retrieves all the indexes (keys) in the database for this enmap, even if they aren't fetched.
Kind: instance property of Enmap
Returns: array.
- Array of all indexes (keys) in the enmap, cached or not.
number
Generates an automatic numerical key for inserting a new value. This is a "weak" method, it ensures the value isn't duplicated, but does not guarantee it's sequential (if a value is deleted, another can take its place). Useful for logging, but not much else.
Kind: instance property of Enmap
Returns: number
- The generated key number.
Example
Enmap
Sets a value in Enmap.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
Required. The key of the element to add to The Enmap.
val
*
Required. The value of the element to add to The Enmap. If the Enmap is persistent this value MUST be stringifiable as JSON.
path
string
null
Optional. The path to the property to modify inside the value object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Example
*
Retrieves a key from the enmap. If fetchAll is false, returns a promise.
Kind: instance method of Enmap
Returns: *
- The value for this key.
Param
Type
Default
Description
key
string
| number
The key to retrieve from the enmap.
path
string
null
Optional. The property to retrieve from the object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Example
Enmap
Fetches every key from the persistent enmap and loads them into the current enmap value.
Kind: instance method of Enmap
Returns: Enmap
- The enmap containing all values.
Enmap
| *
Force fetch one or more key values from the enmap. If the database has changed, that new value is used.
Kind: instance method of Enmap
Returns: Enmap
| *
- The Enmap, including the new fetched values, or the value in case the function argument is a single key.
Param
Type
Description
keyOrKeys
string
| number
| Array.
A single key or array of keys to force fetch from the enmap database.
Enmap
Removes a key or keys from the cache - useful when disabling autoFetch.
Kind: instance method of Enmap
Returns: Enmap
- The enmap minus the evicted keys.
Param
Type
Description
keyOrArrayOfKeys
string
| number
| Array.
A single key or array of keys to remove from the cache.
Function called whenever data changes within Enmap after the initial load. Can be used to detect if another part of your code changed a value in enmap and react on it.
Kind: instance method of Enmap
Param
Type
Description
cb
function
A callback function that will be called whenever data changes in the enmap.
Example
Promise.
Shuts down the database. WARNING: USING THIS MAKES THE ENMAP UNUSEABLE. You should only use this method if you are closing your entire application. Note that honestly I've never had to use this, shutting down the app without a close() is fine.
Kind: instance method of Enmap
Returns: Promise.
- The promise of the database closing operation.
Enmap
Modify the property of a value inside the enmap, if the value is an object or array. This is a shortcut to loading the key, changing the value, and setting it back.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Description
key
string
| number
Required. The key of the element to add to The Enmap or array. This value MUST be a string or number.
path
string
Required. The property to modify inside the value object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
val
*
Required. The value to apply to the specified property.
Enmap
Push to an array value in Enmap.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
Required. The key of the array element to push to in Enmap. This value MUST be a string or number.
val
*
Required. The value to push to the array.
path
string
null
Optional. The path to the property to modify inside the value object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
allowDupes
boolean
false
Optional. Allow duplicate values in the array (default: false).
Example
Enmap
Push to an array element inside an Object or Array element in Enmap.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
Required. The key of the element. This value MUST be a string or number.
path
string
Required. The name of the array property to push to. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
val
*
Required. The value push to the array property.
allowDupes
boolean
false
Allow duplicate values in the array (default: false).
Enmap
Executes a mathematical operation on a value and saves it in the enmap.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
The enmap key on which to execute the math operation.
operation
string
Which mathematical operation to execute. Supports most math ops: =, -, *, /, %, ^, and english spelling of those operations.
operand
number
The right operand of the operation.
path
string
null
Optional. The property path to execute the operation on, if the value is an object or array.
Example
Enmap
Increments a key's value or property by 1. Value must be a number, or a path to a number.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
The enmap key where the value to increment is stored.
path
string
null
Optional. The property path to increment, if the value is an object or array.
Example
Enmap
Decrements a key's value or property by 1. Value must be a number, or a path to a number.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
The enmap key where the value to decrement is stored.
path
string
null
Optional. The property path to decrement, if the value is an object or array.
Example
*
Returns the specific property within a stored value. If the key does not exist or the value is not an object, throws an error.
Kind: instance method of Enmap
Returns: *
- The value of the property obtained.
Param
Type
Description
key
string
| number
Required. The key of the element to get from The Enmap.
path
string
Required. The property to retrieve from the object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
*
Returns the key's value, or the default given, ensuring that the data is there. This is a shortcut to "if enmap doesn't have key, set it, then get it" which is a very common pattern.
Kind: instance method of Enmap
Returns: *
- The value from the database for the key, or the default value provided for a new key.
Param
Type
Default
Description
key
string
| number
Required. The key you want to make sure exists.
defaultValue
*
Required. The value you want to save in the database and return as default.
path
string
null
Optional. If presents, ensures both the key exists as an object, and the full path exists. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Example
boolean
Returns whether or not the key exists in the Enmap.
Kind: instance method of Enmap
Param
Type
Default
Description
key
string
| number
Required. The key of the element to add to The Enmap or array. This value MUST be a string or number.
path
string
null
Optional. The property to verify inside the value object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Example
boolean
Returns whether or not the property exists within an object or array value in enmap.
Kind: instance method of Enmap
Returns: boolean
- Whether the property exists.
Param
Type
Description
key
string
| number
Required. The key of the element to check in the Enmap or array.
path
*
Required. The property to verify inside the value object or array. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Enmap
Deletes a key in the Enmap.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
Required. The key of the element to delete from The Enmap.
path
string
null
Optional. The name of the property to remove from the object. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Delete a property from an object or array value in Enmap.
Kind: instance method of Enmap
Param
Type
Description
key
string
| number
Required. The key of the element to delete the property from in Enmap.
path
string
Required. The name of the property to remove from the object. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
Deletes everything from the enmap. If persistent, clears the database of all its data for this table.
Kind: instance method of Enmap
null
Completely destroys the entire enmap. This deletes the database tables entirely. It will not affect other enmap data in the same database, however. THIS ACTION WILL DESTROY YOUR DATA AND CANNOT BE UNDONE.
Kind: instance method of Enmap
Enmap
Remove a value in an Array or Object element in Enmap. Note that this only works for values, not keys. Complex values such as objects and arrays will not be removed this way.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Default
Description
key
string
| number
Required. The key of the element to remove from in Enmap. This value MUST be a string or number.
val
*
Required. The value to remove from the array or object.
path
string
null
Optional. The name of the array property to remove from. Can be a path with dot notation, such as "prop1.subprop2.subprop3". If not presents, removes directly from the value.
Enmap
Remove a value from an Array or Object property inside an Array or Object element in Enmap. Confusing? Sure is.
Kind: instance method of Enmap
Returns: Enmap
- The enmap.
Param
Type
Description
key
string
| number
Required. The key of the element. This value MUST be a string or number.
path
string
Required. The name of the array property to remove from. Can be a path with dot notation, such as "prop1.subprop2.subprop3"
val
*
Required. The value to remove from the array property.
Array
Creates an ordered array of the values of this Enmap. The array will only be reconstructed if an item is added to or removed from the Enmap, or if you change the length of the array itself. If you don't want this caching behaviour, use Array.from(enmap.values())
instead.
Kind: instance method of Enmap
Array.
Creates an ordered array of the keys of this Enmap The array will only be reconstructed if an item is added to or removed from the Enmap, or if you change the length of the array itself. If you don't want this caching behaviour, use Array.from(enmap.keys())
instead.
Kind: instance method of Enmap
*
| Array.
Obtains random value(s) from this Enmap. This relies on array.
Kind: instance method of Enmap
Returns: *
| Array.
- The single value if count
is undefined, or an array of values of count
length
Param
Type
Description
[count]
number
Number of values to obtain randomly
*
| Array.
Obtains random key(s) from this Enmap. This relies on keyArray
Kind: instance method of Enmap
Returns: *
| Array.
- The single key if count
is undefined, or an array of keys of count
length
Param
Type
Description
[count]
number
Number of keys to obtain randomly
Array
Searches for all items where their specified property's value is identical to the given value (item[prop] === value
).
Kind: instance method of Enmap
Param
Type
Description
prop
string
The property to test against
value
*
The expected value
Example
*
Searches for a single item where its specified property's value is identical to the given value (item[prop] === value
), or the given function returns a truthy value. In the latter case, this is identical to Array.find().
All Enmap used in Discord.js are mapped using their `id` property, and if you want to find by id you should use the `get` method. See [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map/get) for details.
Kind: instance method of Enmap
Param
Type
Description
propOrFn
string
| function
The property to test against, or the function to test with
[value]
*
The expected value - only applicable and required if using a property for the first argument
Example
Example
string
| number
Searches for the key of a single item where its specified property's value is identical to the given value (item[prop] === value
), or the given function returns a truthy value. In the latter case, this is identical to Array.findIndex().
Kind: instance method of Enmap
Param
Type
Description
propOrFn
string
| function
The property to test against, or the function to test with
[value]
*
The expected value - only applicable and required if using a property for the first argument
Example
Example
boolean
Searches for the existence of a single item where its specified property's value is identical to the given value (item[prop] === value
).
Do not use this to check for an item by its ID. Instead, use `enmap.has(id)`. See [MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map/has) for details.
Kind: instance method of Enmap
Param
Type
Description
prop
string
The property to test against
value
*
The expected value
Example
number
Removes entries that satisfy the provided filter function.
Kind: instance method of Enmap
Returns: number
- The number of removed entries
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
Object
Value to use as this
when executing function
Enmap
Identical to Array.filter(), but returns a Enmap instead of an Array.
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
Object
Value to use as this
when executing function
Array
Identical to Array.filter().
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
Object
Value to use as this
when executing function
Array.
Partitions the collection into two collections where the first collection contains the items that passed and the second contains the items that failed.
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
*
Value to use as this
when executing function
Example
Array
Identical to Array.map().
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function that produces an element of the new array, taking three arguments
[thisArg]
*
Value to use as this
when executing function
boolean
Identical to Array.some().
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
Object
Value to use as this
when executing function
boolean
Identical to Array.every().
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to test (should return a boolean)
[thisArg]
Object
Value to use as this
when executing function
*
Identical to Array.reduce().
Kind: instance method of Enmap
Param
Type
Description
fn
function
Function used to reduce, taking four arguments; accumulator
, currentValue
, currentKey
, and enmap
[initialValue]
*
Starting value for the accumulator
Enmap
Creates an identical shallow copy of this Enmap.
Kind: instance method of Enmap
Example
Enmap
Combines this Enmap with others into a new Enmap. None of the source Enmaps are modified.
Kind: instance method of Enmap
Param
Type
Description
...enmaps
Enmaps to merge
Example
boolean
Checks if this Enmap shares identical key-value pairings with another. This is different to checking for equality using equal-signs, because the Enmaps may be different objects, but contain the same data.
Kind: instance method of Enmap
Returns: boolean
- Whether the Enmaps have identical contents
Param
Type
Description
enmap
Enmap to compare with
Migrates an Enmap from version 3 or lower to a Version 4 enmap, which is locked to sqlite backend only. This migration MUST be executed in version 3.1.4 of Enmap, along with appropriate providers. See https://enmap.evie.codes/install/upgrade for more details.
Kind: static method of Enmap
Array.
Initialize multiple Enmaps easily.
Kind: static method of Enmap
Returns: Array.
- An array of initialized Enmaps.
Param
Type
Description
names
Array.
Array of strings. Each array entry will create a separate enmap with that name.
options
Object
Options object to pass to the provider. See provider documentation for its options.
Example
Additional options for the enmap. See for details.
Defaults to deep. Determines how objects and arrays are treated when inserting and retrieving from the database. See for more details on this option.