Is it mandatory to keep curl-access.txt ...

Up to Discussions in Ask-the-Expert

This Question is Answered

15 "helpful" answers available (3 pts)

1 2 Previous Next

20 Replies Last post: May 27, 2008 5:06 AM by URPradhan

Is it mandatory to keep curl-access.txt ?

May 23, 2008 12:05 AM

URPradhan 166 posts since
Mar 6, 2008

Hi Friends

As per the Curl documentation "Any Web site to which your applet is going to submit a request must have a curl-access.txt that grants the applet access".

And I'm posting this thread to confirm for the below specific situation.

Lets say I know a web server accepting requests from ANY client using HTTP GET method and it does not care in which language the client has written. You may think of a typical GET method query to the server like "http://www.example.com/test1/myscript.php?username=xyz&password=abc". And I'm getting the server response if I'm using any client (except Curl applets) or may be directly from browser.

But, if my client is a Curl applet, then I should also get the response from the server, because the server has given permission to public for that interface and its server admin's responsibility to put security measures at his end.

Then, in the above situation..... Is it still mandatory to keep a curl-access.txt file at the root of the server , ie, http://www.example.com/ ???
If yes, and the server is not owned by me (I CAN NOT put any file at server root) then how can I write a Curl application to request for data from the server ? where as client applications written in other languages, can post a query to server easily without any restriction.

Do not you think its a self restriction to Curl applets for public servers ?

Need your valuable input.

//Thank you

Tags: curl-access.txt

friedger 108 posts since
Jan 13, 2008

1. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 12:43 AM

The documentation "Allowing Applets to Access Files on a Web Site" explains nicely why you don't want to allow unprivileged Curl applets to allow web site silently.

Has the server a crossdomain.xml? Or the user has to give privileges to the applet.

Sure, if you compare Curl with desktop applications the Curl security model is too restricitve. However, the strength of Curl security is that you can trust an unprivileged applet in the same way as you can trust a static HTML site.

Friedger

URPradhan 166 posts since
Mar 6, 2008

2. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 2:56 AM

in response to: friedger

Hi friedger

I had read the topic "Allowing Applets to Access Files on a Web Site" before posting the thread. But unfortunately lets say that site has no "crossdomain.xml" file too. But my concern is ...

If the web site is not putting any restrictions for read-only queries to its files/data (only allowing users to query through its proper secure interfaces) then why Curl imposes the restrictions to self. Generally restrictions kept at server side, but here the client is putting himself in trouble and sitting aside to others and axing to self. In this situation, I really do not see any valid reason for this restriction within RTE.

Lets say Google is offering some services to public and anyone can make use of those services using some defined interfaces, then in that case, will we ask google to put a curl-access.txt or crossdomain.xml file in their server root ?

I'm sorry if my requirement is not valid (plz correct me) and I have not understood it properly. But I'm confused with this feature/restriction.

//Thank you

fukuta 118 posts since
Oct 17, 2007

3. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 3:40 AM

in response to: URPradhan

Maybe its restriction is mainly for the sake of users security.
Curl doesn't consider the applet safe even if its developer believes the applet connects only to safe servers.
Whether the server allows accesses to itself or not, an applet needs curl-access.txt if it is unprivileged.

friedger 108 posts since
Jan 13, 2008

4. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 4:12 AM

in response to: fukuta

A workaround would be to mirror the webservice on a server that you own and let the applet access your server.

Or ask google or whoever to add a crossdomain.xml or curl-access.txt file.

Friedger

URPradhan 166 posts since
Mar 6, 2008

5. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 4:31 AM

in response to: friedger

So its mandatory to put curl-access.txt file in a webserver to which a curl applet will make query. There is no way to escape from this. And unfortunately I can not afford your first suggestion as I do not own any web server. Secondly sites like Google or similar will not listen to me if I'll ask them to keep a curl-access.txt file on their server root. They may argue that, lets say tomorrow another language XYZ, will evolve then will we have to put a XYZ-access.txt file on our server, which is unacceptable.

Thank you all for your valuable inputs which helped me to learn this feature.

friedger 108 posts since
Jan 13, 2008

6. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 5:14 AM

in response to: URPradhan

My understanding is that you can also use a crossdomain.xml file. That is not Curl specific. Just give it a go and ask them.

Friedger

RMH 45 posts since
Feb 17, 2008

7. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 5:22 AM

Hi URPradhan,

There is actually some excellent reasoning behind the curl-access.txt security mechanism - in fact it makes so much sense that Adobe Flash/Flex copied the mechanism and uses it as well. I've asked our engineers to jump on and answer this question providing a clear explanation. There are ways to allow a unprivileged applet access to domains other than its domain of origin. I'll let the engineer explain how that works.

Richard

URPradhan 166 posts since
Mar 6, 2008

8. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 5:43 AM

in response to: RMH

Thank you RHM by hearing such words from you.

But it certainly confuses a developer like me. And I've googled it for "crossdomain.xml" file and seems that quite a few sites are now keeping a copy of this in their server root. Having said that its too difficult to convince a server's owner to keep a copy of this file.

Another aspect of this is, as a user I know to which site and to what for I'm making a query. And I can have trust/faith upon that server for their APIs. So in that case I should be allowed to query to the server. Again I'm putting the case as ... I'll host my curl applet in my site and that curl applet will make query to another server which hosts the APIs. Here I can understand the logic of putting a curl-access.txt file in the domain of origin (at my site) but not for the server which hosts the APIs.

//Thank you

RMH 45 posts since
Feb 17, 2008

9. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 6:33 AM

in response to: URPradhan

I spent some time reading the documentation at "Allowing Applets to Access Files on a Web Site". It does take a stab at explaining why Curl uses curl-access.txt but I'll try to explain it more clearly.

Lets say that the RTE never required a curl-access.txt file be placed on web servers you want to access for data; that a Curl applet could access any web server it wants "silently" which means without asking the end user permission. In that case, Curl applications could potentially access servers behind your firewall provided you download them to your desktop browser inside your company’s int*RA*net. You don't want that, so a Curl applet has to check any domain it access to see if there is a curl-access.txt file before allowing access to that domain. If the web servers running inside your firewall do not have a curl-access.txt file then the Curl applet cannot access them. Makes sense right? You don't want a Curl tic-tack-toe game you downloaded from xyz-games.com to have unrestricted access to your internal servers do you?

While the malicious applet that invades your internal web servers is more the exception than the rule - its never-the-less a huge security concern. So to avoid the situation Curl created the curl-access.txt security policy, which says that a Curl applet can only access web sites that have the curl-access.txt file. Sites that want to give you access can put a curl-access.txt on the root of their web site; web servers that don't want to provide arbitrary applications with access (such as internal web servers) can simply not put a curl-access.txt on their root directory. This solution is seen by the security community as being very effective. In fact, Adobe copied the solution a couple of years ago and added it to their Flash player.

Ideally all web sites that have data accessible to outside parties (e.g. Google) would put up a curl-access.txt file or Adobe's crossdomain.txt file (Curl can use either one I believe), but not everyone is going to do that and its not going to be easy for "Joe programmer" to convince a big site like Google to do it. In these cases, you can ask the end user to grant access to a web site explicitly.

Curl provides this thing called the security dialog. When your Curl applet (unprivileged) needs to access a URL that doesn't have a curl-access.txt file you can catch the security exception and call the "choose-location" procedure. This will pop up a security dialog that requires the user to explicitly give the RTE permission to access that URL from the applet. You could also do this without checking for curl-access.txt first by simply calling the choose-location method before attempting to access a URL. That way you can be proactive rather than reactive. Either way works.

All this is explained in the section “Allowing Applets to Access Files on a Web Site” and the section "Dialog Procedures". The choose-location procedure can be found in the API documentation.

Hopefully this explanation makes sense and provides you with a mechanism to get around the curl-access.txt in a way that is safe for your end users. You could, as an alternative, deploy the application as privileged but that alternative is so fraught with security issues we insist that you get a special license to do it - if you could create fully privileged applications without constraint than Curl would be a hackers delight in the same way that Adobe AIR is a hacker's delight.

I might create a video that explains this more clearly than the documentation currently does. The documentation is good, but you have to read it a couple of times to pick up the things I addressed here.

rhh 29 posts since
Oct 12, 2007

10. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 11:56 AM

This is a very tricky subject to think about. And on top of that, it's obviously inconvenient not to be able to write an applet that accesses any resource without restriction, and it's very hard to understand why a resource that (to us) is "obviously public" can't be accessed by any Curl applet.

RMH and Friedger have already explained the basic reasons for this policy. As they have already explained, the basic problem is that an applet that is executing on a machine inside a firewall cannot be allowed to access servers both inside and outside the firewall unless the applet is totally trusted, because such an applet could read confidential information from inside the firewall and send it to a server outside the firewall. Because of this reason, every system for safe execution of untrusted applets from the internet, starting with Java and continuing with JavaScript, Flash, Curl, and Silverlight, has had to implement a restriction on such accesses.

Unfortunately, there is no bulletproof way for an applet to look at a server's domain name or IP address and determine whether that server is inside or outside of a firewall. So the original Java policy was that an applet could only communicate with the server that it was loaded from. As long as an applet can only talk to one server, it cannot possibly steal information from one server and send it to a different server. This "only call home" policy has also been adopted by every subsequent RIA platform as the basic communication security policy for safe applet execution.

At Curl, we take security very seriously, so we had to implement a policy like this too. However, we recognize the value, for mashups and other purposes, of having an applet that can communicate with multiple servers. We worked hard to think of a safe way to extend the "only call home" policy so that there could be applets that still execute in safe mode but are able to contact multiple servers. The curl-access.txt mechanism was the result of this work, and as RMH mentioned, this idea was later copied in Flash's crossdomain.xml mechanism. The idea is that the only safe way to know that a server is publicly accessible is for the server to label itself as publicly accessible. A server can do this by having one of these files on it. We recognize the inconvenience of having to persuade a server administrator to deploy one of these files, but any mechanism for considering a server to be a public server without one of these files would open up a security hole allowing the theft of information as I described above.

(Edit: I removed a mistaken comment about how a user can explicitly authorize access to URLs on servers that don't have a curl-access.txt file.)

-Bert

RMH 45 posts since
Feb 17, 2008

11. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 11:41 AM

in response to: rhh

@rhh,

Thank you very much for taking the time to address this issue. Your explanation was excellent.

I think you can use the choose-locaiton prodecure to launch a dialog box that allows the end user to input a specific URL to be allowed. I've asked a couple of other engineers for clarification. You are right that you cannot set this up via the RTE control panel which is different than what I'm talking about. But as you said, this stuff gets a little complicated so hopefully the other engineers will be able to clarify.

Richard

rhh 29 posts since
Oct 12, 2007

12. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 11:57 AM

in response to: RMH

Sorry, I read your previous post too quickly. As soon as I saw "security dialog" I decided you were talking about the Curl Control Panel. I agree that choose-location has the capability that you're talking about. I've edited my previous post accordingly.

-Bert

wbardwell 75 posts since
Oct 31, 2007

13. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 12:58 PM

in response to: rhh

Some clarifications:

choose-location lets the user enter a single URL that the applet an then access, it does not allow access to the whole website, just hat one URL.

As Bert noted the security stuff in the control panel allows an applet from the location run privileged (if they have a pro license), and then the applet can access anything.

Also I would note that the issue that curl-access.txt is trying to solve, is not just for servers on a user's intranet, it is also for any web server that requires log-ins, because
an applet could potentially access a server that the user has already logged-in to with the user's credentials, and, for example transfer money from your bank account.

We have considered adding one of the following additional mechanisms to get around curl-access.txt files, but haven't done so yet, for the reasons listed below.
1) Have a way for the user to say "this applet can access any http: url", which would make an applet partially privileged, but still restricted from accessing the local disk. But the trouble is that users might not realize how much power they are granting the applet, and so they might as well just grant it full privilege.
2) Have a way for the user to say "this applet can access this web server", which is reasonable, but an applet making the user enter more than one or two of these would be very annoying.
3) Have a way for a user to say "the following web servers are public, allow any applet to access them", which would also be a bit confusing and annoying to enter more than a few of them.

Thus, we haven't come up with a better solution than ask the web server owner to put curl-access.txt or crossdomain.xml files down, or run the applet privileged. If a developer signs an applet, that makes it a less annoying experience for the user to allow it to run privileged. And we might still implement one of the above in the future.

If anyone has an idea for a solution to this issue that meets the following criteria we would love to hear it.

1) The question that the user is asked is easy to understand. (Like the one for a privileged applet "Do you trust this applet with your life.")

2) The user won't be asked again and again.

3) The result solves more than one specific applet's need.

cbarber 124 posts since
Sep 27, 2007

14. Re: Is it mandatory to keep curl-access.txt ? May 23, 2008 3:35 PM

in response to: wbardwell

I am probably not a representative user, but I would love to see something like #3. I routinely block javascript and cookies except on explicitly allowed sites. One reason this isn't too annoying is that FireFox extensions provide easy ways to enable and disable sites without having to bring up a dialog and type, but even if I did have to type a few sites I think this would be a big help in making use of public data sources lacking public access curl-access.txt or cross-domain.xml files.

1 2 Previous Next

This Question is Answered

Is it mandatory to keep curl-access.txt ?

May 23, 2008 12:05 AM

Actions

More Like This