[Filebrowser] Add get delegation token logic for secure hadoop (#3301) (Related with: #3324 ) #3449
Conversation
fix recursive call of delegation token
|
Hi @SoniaComp and thanks for contributing again :-) |
|
This PR is stale because it has been open 45 days with no activity and is not labeled "Prevent stale". Remove "stale" label or comment or this will be closed in 10 days. |
|
I use this bug fix in my secure hadoop environment successfully, so I'm sure that this bug fix work well. |
|
@ranade1 friendly reminder from our contributor here, can you take a look at this PR? |
|
@SoniaComp, The "cachetools" library you are utilizing is not thread-safe. See https://cachetools.readthedocs.io/en/latest/ |
|
@ranade1 Thank you for your comment! I will change that library with better one. |
|
I tested this code and confirmed that it works properly after fixing the bug. |
| @@ -32,12 +33,13 @@ | |||
| import time | |||
| import urllib.request, urllib.error | |||
|
|
|||
| from django.core.cache import caches | |||
There was a problem hiding this comment.
�Requesting a delegation token every time using filebrowser can put a stress on the hadoop namenode. So I used cache.
| CACHES[CACHES_WEBHDFS_DELEGATION_TOKEN_KEY] = { | ||
| 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', | ||
| 'LOCATION': CACHES_WEBHDFS_DELEGATION_TOKEN_KEY, | ||
| 'TIMEOUT': desktop.conf.KERBEROS.REINIT_FREQUENCY |
There was a problem hiding this comment.
Kerberos tickets are renewed periodically, so I use Timeout option that the cache expires accordingly.
| if self._security_enabled: | ||
| token = cache.get(self.user, None) | ||
| if not token: | ||
| token = self.get_delegation_token(self.user) |
There was a problem hiding this comment.
To use the impersonation function to grant permission to each user, you must use a delegation token. (reference: https://blog.cloudera.com/hadoop-delegation-tokens-explained/)
| def get_delegation_token(self, renewer): | ||
| """get_delegation_token(user) -> Delegation token""" | ||
| # Workaround for HDFS-3988 | ||
| if self._security_enabled: |
There was a problem hiding this comment.
This problem was fixed in hadoop 2.6 version. (reference: https://issues.apache.org/jira/browse/HDFS-3988)
|
@SoniaComp Thanks! @ranade1 Can you have another look? |
|
Thank you for your reviews! Are there any parts in this PR that could be improved further? |
|
Hello @SoniaComp , Thank you for putting code changes. I have specific questions.
|
|
Hi! I appreciate for the good questions! 😊
|
|
Do I need more test for this code? |
|
@SoniaComp thank you. @ranade1 @amitsrivastava Can we please take a moment to see if we can get this PR merged or if additional changes/tests are required? |
|
This PR is stale because it has been open 45 days with no activity and is not labeled "Prevent stale". Remove "stale" label or comment or this will be closed in 10 days. |
|
This is needed. |
|
This PR is stale because it has been open 45 days with no activity and is not labeled "Prevent stale". Remove "stale" label or comment or this will be closed in 10 days. |
|
@SoniaComp |
What changes were proposed in this pull request?
I am working on integrating Hue with SecureHadoop as a ProxyUser.
I found that the behavior of ProxyUser in WebHDFS FileBrowser needs to be enhanced.
When "security_enabled" is true, only "read_url" method use delegation token.
I want to modify the code
so that a user who has been impersonated from Hue(a proxy user) can issue a Delegation Token when using WebHDFS.
(#3323)