Python is Not a Configuration File Format
There is a large thread in reddit about using Python as configuration file format. I want to be clear about that:
DON'T DO THAT, UNLESS YOU HAVE A VERY GOOD REASON.
If you need to ask if it's a good idea, then you don't have a good reason. If you are sure that you have a good reason, then maybe you have a good reason.
There are many reasons for it, but I will explore just two, then offer a suggestion.
Python is read-only, and configuring is not programming.
Sure, it's easy to use python as a config file. You just import the thing, and there you go, all the data is there. But now your configuration syntax is a general purpose language that can do things like pop up dialogs when you parse it.
Your config can now depend on the whole internet, the hardware, the weather, and interactive input. Powerful? Sure. Good idea? Maybe, sometimes. But your application is now not able to configure itself.
If your application wants to store any kind of setting, it won't be able to. So most interactive, desktop apps, just should not use python for this, ever.
But what about non-interactive tools? Well, using python means that other tools can't write to the config file, either, which makes the tool less powerful. The power to have tools use tools is one of the cornerstones of modern computing, and you just cut your app off that ecosystem. Depending on what language the tool uses it may not even be able to parse your config file.
And what happens when someone is told "use this config fragment to achieve X"? Well, odds are, if the recipient has done anything that takes advantage of using python as a config format, then the fragment will not work. It would be like doing copy/paste from random code in github into your own program and expecting it to work.
So, you can't write to it from the app, you can't get configuration tips from the internet, you can't use other tools to modify config files, and other tools have a hard time parsing your files.
Also, it means that to handle the general case of configuring your app, you need a programmer. That is almost certainly overkill. Very few apps need that kind of thing. If your app can only be configured by programmers, you may have failed at making a good app (exceptions exist).
And what's the advice? Well, the advice is "don't do that" and the corollary is "configure using data, not code". use INIs, or XML, or YAML, or JSON, or plain text files, or whatever. But not code.
PS: My latest project, Nikola uses python as a configuration language. I thought I had a good reason. I didn't.
> Python is Not a Configuration File Format
But a subset of python syntax is.
Yes. But if you say "the syntax is python" then it's not a subset.
Aren't you falling prey to the “if it isn't forbidden, then it's mandatory” fallacy?
The fact that you can use code to configure the app doesn't mean that you must use it. You can say “The syntax is this subset of Python. Actually, that's a lie: the syntax is Python, and you can use its full power if you need to. But please make sure you really need to before you venture there, because it's dangerous to dwell outside boundaries of the subset: you will be unable to save your settings from the app or from third party tools, your code may trigger additional dependencies&hellip as a rule of thumb, don't use full Python in the config file unless you can name at least three things than can go wrong, and you know you can live with them. If you can't, you probably don't know enough about the problem to assess the risk.”
Python as a configuration language is a power tool. As with all other power tools, they can cause unwanted holes to appear in your body if you are not skilled in their use. But that's no reason to keep them away from those who are skilled.
Short answer: No.
Longer answer: I am very careful with my power tools. I don't leave them in park benches, plugged, with notes saying "only use if you really need the power".
Often, python-as-settings happens because it's easier and saves 4 lines of ConfigParser boilerplate. If the task really needs the power, then by all means, use powertools.
Besides, the "but python is more powerful" doesn't address any of the arguments against it I described in the post. A chainsaw is powerful, but makes for a lousy nail clipper.
if so, tell that to twistedmatrix guys, that .tac files they use in twisted - was wrong choice. choice which do not require additional parsing, which is specific for python, which is really simplier than xml (parsing xml is always expensive). or... maybe twisted is too small and too young framework as for You? ;)
if You are reading some configuration in python application - what would be output produced from this configuration? most probably - python variables... so at end You will convert your configuration into ... python representation.
btw. what You have in mind by writing "your application is now not able to configure itself". what this has connected with configuration file format? do You know other method of reading configuration from .py file than importing it?
If twisted jumped off a bridge, would you jump too? ;-)
I am not advocating XML, specifically, there are lots of other formats to store data.
When I write "your application is now not able to configure itself" I mean exactly that. Your application can't save data reliably into a configuration file whose syntax is arbitrary python code. If you can, congratulations, you have a program that solves the halting problem.
And no, if you read a config file from python, the product is not "a python representation". It's an in-memory representation valid for whatever VM you are using to run your program.
I don't understand where you're going with that about " do You know other method of reading configuration from .py file than importing it?" so I'll abstain.
Sorry, but what does "python script writing another script" has with halting problem?
It's a program that modifies another program (the config file). Unless you are ok with your configuration file not halting, then you have to solve the halting problem.
I use YAML for my config files and agree with a lot of your points, but the halting problem is not a very strong argument that python config files would be unreliable on a practical level. There are plenty of cases where any config file format would theoretically cause problems in very specific but unlikely cases.
For example, let's say your config parser supports an arbitrary length list for one of its settings, or even will just read an arbitrary amount of sections of the config file. If so depending on the operating system I could easily setup a scenario where your program reads an "infinite" config file and either crashes due to lack of memory or otherwise has undefined behavior. That doesn't mean that config files shouldn't ever allow arbitrary length lists in them though, it's only something you have to worry about if infinite config files are actually a realistic problem that you'll have to deal with (such as someone using them as an exploit).
Well, infinite config files are only a problem if you have infinite disks :-)
Yes, the halting problem is somewhat tongue-in-cheek, but the infinite potential complexity of programs considered as config files is a very real concern.
Even config files that are data can fall in that trap (example: sendmail)
most configuration files start little, so there's really no reason to kick in the power of a real programming language. but sooner or later, you need to add conditional configuration ("if windows:... elif linux:..."), and then it might make sense to split your configuration over several files which include/import a common one, and the list of features grows longer.
long story short - at some point, you will need a powerful language (maybe not turing complete, but surely more powerful than key=value), so why invent a new DSL when you can just use python?
No, not every app needs that. And why not do that? Because of the things the article says, among others.
the question is -- when you realize you need more than KEY=VALUE in your configuration, what should you do?
I, at least, understand your position, having being there myself, but now I agree with Roberto. I'm not able to answer your question, unless given a concrete example, like yours: "if windows:... elif linux:...", in that case, I would read/write my settings in one of the following ways:
setting = value_for_win
setting = value_for_linux
setting.linux = value_for_linux
setting.windows = value_for_win
linux = value_for_linux
windows = value_for_win
I would not suggest JSON as configuration file format, because it is lacking support for comments. Just sayin :)
Seconded. My favorite text editor uses JSON for most of its settings, and I'm always getting parsing errors because a "one last comma" that should not be there, or a missed one, or "True" instead of "true", etc.
Great format for data interchange, not good enough for human editable content.
Good points both.
My rule of thumb for configuration files:
If I cannot figure it out how to do something with .ini files, I should not be doing that anyway, and should be looking for a simpler solution that indeed works with .ini files :-D.
Agreed. However, I think there is one thing to be said in favor of using Python as configuration, or rather, one prominent reason that many apps choose to use it: it's dead simple to use. Compare "import settings" to the whole ConfigParser song and dance you're required to use for that format, and for an early stage project using Python starts to look really appealing.
Are there any good modules that can make this easer? Probably. Are they well known? Maybe not (I don't know of any, nor does the Hitchhiker's Guide). I think if we can solve this hurdle, or find and promote a package that already does, we can begin to effectively combat this bad practice; until then, and with Django and Twisted setting bad examples (in my opinion, and I believe in yours), it's unlikely to change.
I tend to use a small wrapper around SafeConfigParser which is very inefficient (doesn't matter in context) but gets the work done:
As with any binary argument, I think the answer is "both".
The actual danger is expecting to be able to use "import" to retrieve configuration settings. It's a convenience that leads to tears. I suspect most Django core developers would agree (e.g. "import settings" is a source of circref problems and an inability to put more than one Django configuration into the same Python process).
Instead, it seems saner to put true configuration in a non-importable format (e.g. ini). Then have your code read that file, and operate *against the result* to do conditionalized things, e.g.:
adict = read_ini_file(the_ini_file)
In this way you can get the benefit of simplicity of pure-ini-syntax when you don't need conditionals, but you have a place to stick Python code to do conditionals and looping where necessary. You can also always create a "settings.py" that reads the ini at module scope and injects the result into module globals, so if you really *must* "import settings" you can do it.. you just can't go the other way around and turn a system that expects settings to be importable into one that does not.