Skip to main content

Ralsina.Me — Roberto Alsina's website

Python is Not a Configuration File Format

There is a large thread in red­dit about us­ing Python as con­fig­u­ra­tion file for­mat. I want to be clear about that:

DON'T DO THAT, UN­LESS YOU HAVE A VERY GOOD REA­SON.

If you need to ask if it's a good idea, then you don't have a good rea­son. If you are sure that you have a good rea­son, then maybe you have a good rea­son.

There are many rea­sons for it, but I will ex­plore just two, then of­fer a sug­ges­tion.

Python is read­-on­ly, and con­fig­ur­ing is not pro­gram­ming.

Sure, it's easy to use python as a con­fig file. You just im­port the thing, and there you go, all the da­ta is there. But now your con­fig­u­ra­tion syn­tax is a gen­er­al pur­pose lan­guage that can do things like pop up di­alogs when you parse it.

Your con­fig can now de­pend on the whole in­ter­net, the hard­ware, the weath­er, and in­ter­ac­tive in­put. Pow­er­ful? Sure. Good idea? May­be, some­times. But your ap­pli­ca­tion is now not able to con­fig­ure it­self.

If your ap­pli­ca­tion wants to store any kind of set­ting, it won't be able to. So most in­ter­ac­tive, desk­top app­s, just should not use python for this, ev­er.

But what about non-in­ter­ac­tive tool­s? Well, us­ing python means that oth­er tools can't write to the con­fig file, ei­ther, which makes the tool less pow­er­ful. The pow­er to have tools use tools is one of the cor­ner­stones of mod­ern com­put­ing, and you just cut your app off that ecosys­tem. De­pend­ing on what lan­guage the tool us­es it may not even be able to parse your con­fig file.

And what hap­pens when some­one is told "use this con­fig frag­ment to achieve X"? Well, odds are, if the re­cip­i­ent has done any­thing that takes ad­van­tage of us­ing python as a con­fig for­mat, then the frag­ment will not work. It would be like do­ing copy­/­paste from ran­dom code in github in­to your own pro­gram and ex­pect­ing it to work.

So, you can't write to it from the ap­p, you can't get con­fig­u­ra­tion tips from the in­ter­net, you can't use oth­er tools to mod­i­fy con­fig files, and oth­er tools have a hard time pars­ing your files.

Al­so, it means that to han­dle the gen­er­al case of con­fig­ur­ing your ap­p, you need a pro­gram­mer. That is al­most cer­tain­ly overkil­l. Very few apps need that kind of thing. If your app can on­ly be con­fig­ured by pro­gram­mer­s, you may have failed at mak­ing a good app (ex­cep­tions ex­ist).

And what's the ad­vice? Well, the ad­vice is "don't do that" and the corol­lary is "con­fig­ure us­ing data, not code". use IN­Is, or XM­L, or YAM­L, or JSON, or plain text files, or what­ev­er. But not code.

PS: My lat­est pro­jec­t, Niko­la us­es python as a con­fig­u­ra­tion lan­guage. I thought I had a good rea­son. I did­n't.

bav / 2012-07-04 05:13:

> Python is Not a Configuration File Format

But a subset of python syntax is.

Roberto Alsina / 2012-07-04 10:11:

Yes. But if you say "the syntax is python" then it's not a subset.

Fede Heinz / 2012-07-04 12:55:

Aren't you falling prey to the “if it isn't forbidden, then it's mandatory” fallacy?

The fact that you can use code to configure the app doesn't mean that you must use it. You can say “The syntax is this subset of Python. Actually, that's a lie: the syntax is Python, and you can use its full power if you need to. But please make sure you really need to before you venture there, because it's dangerous to dwell outside boundaries of the subset: you will be unable to save your settings from the app or from third party tools, your code may trigger additional dependencies&hellip as a rule of thumb, don't use full Python in the config file unless you can name at least three things than can go wrong, and you know you can live with them. If you can't, you probably don't know enough about the problem to assess the risk.”

Python as a configuration language is a power tool. As with all other power tools, they can cause unwanted holes to appear in your body if you are not skilled in their use. But that's no reason to keep them away from those who are skilled.

Roberto Alsina / 2012-07-04 13:11:

Short answer: No.

Longer answer: I am very careful with my power tools. I don't leave them in park benches, plugged, with notes saying "only use if you really need the power".

Often, python-as-settings happens because it's easier and saves 4 lines of ConfigParser boilerplate. If the task really needs the power, then by all means, use powertools.

Besides, the "but python is more powerful" doesn't address any of the arguments against it I described in the post. A chainsaw is powerful, but makes for a lousy nail clipper.

Piotr Skamruk / 2012-07-04 09:12:

if so, tell that to twistedmatrix guys, that .tac files they use in twisted - was wrong choice. choice which do not require additional parsing, which is specific for python, which is really simplier than xml (parsing xml is always expensive). or... maybe twisted is too small and too young framework as for You? ;)

if You are reading some configuration in python application - what would be output produced from this configuration? most probably - python variables... so at end You will convert your configuration into ... python representation.
btw. what You have in mind by writing "your application is now not able to configure itself". what this has connected with configuration file format? do You know other method of reading configuration from .py file than importing it?

Roberto Alsina / 2012-07-04 10:17:

If twisted jumped off a bridge, would you jump too? ;-)

I am not advocating XML, specifically, there are lots of other formats to store data.

When I write "your application is now not able to configure itself" I mean exactly that. Your application can't save data reliably into a configuration file whose syntax is arbitrary python code. If you can, congratulations, you have a program that solves the halting problem.

And no, if you read a config file from python, the product is not "a python representation". It's an in-memory representation valid for whatever VM you are using to run your program.

I don't understand where you're going with that about " do You know other method of reading configuration from .py file than importing it?" so I'll abstain.

para / 2012-07-04 12:20:

Sorry, but what does "python script writing another script" has with halting problem?

Roberto Alsina / 2012-07-04 12:34:

It's a program that modifies another program (the config file). Unless you are ok with your configuration file not halting, then you have to solve the halting problem.

Chris Graham / 2012-07-05 20:07:

I use YAML for my config files and agree with a lot of your points, but the halting problem is not a very strong argument that python config files would be unreliable on a practical level. There are plenty of cases where any config file format would theoretically cause problems in very specific but unlikely cases.

For example, let's say your config parser supports an arbitrary length list for one of its settings, or even will just read an arbitrary amount of sections of the config file. If so depending on the operating system I could easily setup a scenario where your program reads an "infinite" config file and either crashes due to lack of memory or otherwise has undefined behavior. That doesn't mean that config files shouldn't ever allow arbitrary length lists in them though, it's only something you have to worry about if infinite config files are actually a realistic problem that you'll have to deal with (such as someone using them as an exploit).

Roberto Alsina / 2012-07-05 20:15:

Well, infinite config files are only a problem if you have infinite disks :-)

Yes, the halting problem is somewhat tongue-in-cheek, but the infinite potential complexity of programs considered as config files is a very real concern.

Even config files that are data can fall in that trap (example: sendmail)

tomerfiliba / 2012-07-04 10:55:

most configuration files start little, so there's really no reason to kick in the power of a real programming language. but sooner or later, you need to add conditional configuration ("if windows:... elif linux:..."), and then it might make sense to split your configuration over several files which include/import a common one, and the list of features grows longer.

long story short - at some point, you will need a powerful language (maybe not turing complete, but surely more powerful than key=value), so why invent a new DSL when you can just use python?

Roberto Alsina / 2012-07-04 11:01:

No, not every app needs that. And why not do that? Because of the things the article says, among others.

tomerfiliba / 2012-07-04 11:16:

the question is -- when you realize you need more than KEY=VALUE in your configuration, what should you do?

Anonymous Coward / 2012-07-04 12:17:

I, at least, understand your position, having being there myself, but now I agree with Roberto. I'm not able to answer your question, unless given a concrete example, like yours: "if windows:... elif linux:...", in that case, I would read/write my settings in one of the following ways:

----
[Windows]
setting = value_for_win

[Linux]

setting = value_for_linux
----

or...

----
[SomeSection]
setting.linux = value_for_linux
setting.windows = value_for_win
----

or....

[setting]
linux = value_for_linux
windows = value_for_win

beezzik / 2012-07-04 11:54:

I would not suggest JSON as configuration file format, because it is lacking support for comments. Just sayin :)

Anonymous Coward / 2012-07-04 12:22:

Seconded. My favorite text editor uses JSON for most of its settings, and I'm always getting parsing errors because a "one last comma" that should not be there, or a missed one, or "True" instead of "true", etc.

Great format for data interchange, not good enough for human editable content.

Roberto Alsina / 2012-07-04 12:37:

Good points both.

Anonymous Coward / 2012-07-04 12:11:

My rule of thumb for configuration files:

If I cannot figure it out how to do something with .ini files, I should not be doing that anyway, and should be looking for a simpler solution that indeed works with .ini files :-D.

Dan Crosta / 2012-07-04 12:18:

Agreed. However, I think there is one thing to be said in favor of using Python as configuration, or rather, one prominent reason that many apps choose to use it: it's dead simple to use. Compare "import settings" to the whole ConfigParser song and dance you're required to use for that format, and for an early stage project using Python starts to look really appealing.

Are there any good modules that can make this easer? Probably. Are they well known? Maybe not (I don't know of any, nor does the Hitchhiker's Guide). I think if we can solve this hurdle, or find and promote a package that already does, we can begin to effectively combat this bad practice; until then, and with Django and Twisted setting bad examples (in my opinion, and I believe in yours), it's unlikely to change.

Roberto Alsina / 2012-07-04 12:37:

I tend to use a small wrapper around SafeConfigParser which is very inefficient (doesn't matter in context) but gets the work done:

https://code.google.com/p/u...

Chris McDonough / 2012-07-04 16:13:

As with any binary argument, I think the answer is "both".

The actual danger is expecting to be able to use "import" to retrieve configuration settings. It's a convenience that leads to tears. I suspect most Django core developers would agree (e.g. "import settings" is a source of circref problems and an inability to put more than one Django configuration into the same Python process).

Instead, it seems saner to put true configuration in a non-importable format (e.g. ini). Then have your code read that file, and operate *against the result* to do conditionalized things, e.g.:

def config(the_ini_file):
adict = read_ini_file(the_ini_file)
if adict.get('debug'):
...
...

In this way you can get the benefit of simplicity of pure-ini-syntax when you don't need conditionals, but you have a place to stick Python code to do conditionals and looping where necessary. You can also always create a "settings.py" that reads the ini at module scope and injects the result into module globals, so if you really *must* "import settings" you can do it.. you just can't go the other way around and turn a system that expects settings to be importable into one that does not.


Contents © 2000-2020 Roberto Alsina