ngokli Posted June 1, 2017 Report Share Posted June 1, 2017 I installed plastic on a new CentOS 7 server yesterday, and copied the sqlite databases and config files from an old server we are decomissioning. Then, a few hours after starting the plasticsd service, at midnight UTC, I found these error messages in my ssh session console: $ sudo service plasticsd start Starting the Plastic SCM server: OK $ log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.relevant.log.txt] -> [plastic.relevant.log.txt.20170531] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20170531] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.channelcall.log.txt] -> [plastic.channelcall.log.txt.20170531] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 At this point, the log files were cleared, and they continue to accumulate new messages. There are old log files. I couldn't find the errors above in any log file, though (even grepping all of /var/log). I suspect they (luckily) showed up in that terminal because that's where I'd started plasticsd. plasticd is running as the plasticscm user: $ ps aux | grep -i 'plastic\|RSS TTY' USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 16679 0.0 0.0 111788 1476 pts/0 S 19:38 0:00 sudo -u plasticscm -b -n /opt/plasticscm5/server/plasticd --daemon plastic+ 16681 0.0 1.1 1318244 92192 pts/0 Sl 19:38 0:01 plasticd /opt/plasticscm5/server/plasticd.exe --daemon ngokli 16817 0.0 0.0 112652 984 pts/0 S+ 20:00 0:00 grep --color=auto -i plastic\|RSS TTY Directory permissions look okay: $ pwd /opt/plasticscm5/server $ ls -ldh *.log *.log.txt . .. drwxr-xr-x. 7 plasticscm plasticscm 8.0K Jun 1 20:03 . drwxr-xr-x. 7 root root 76 May 31 19:14 .. -rw-r--r--. 1 plasticscm plasticscm 0 May 31 19:14 audit.log -rw-r--r--. 1 plasticscm plasticscm 451K Jun 1 20:14 plastic.channelcall.log.txt -rw-r--r--. 1 plasticscm plasticscm 16M Jun 1 20:14 plastic.debug.log.txt -rw-r--r--. 1 plasticscm plasticscm 642 Jun 1 19:37 plastic.errors.log.txt -rw-r--r--. 1 plasticscm plasticscm 874K Jun 1 20:14 plastic.realtime.requests.log.txt -rw-r--r--. 1 plasticscm plasticscm 8.9M Jun 1 20:14 plastic.relevant.log.txt SELinux is in permissive mode (so it shouldn't block anything): $ sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: permissive Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 I installed plasticscm-complete-5.4.16.918-1.1, to stay as close to my old server as possible (will upgrade soon): $ yum list installed '*plastic*' Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirror.genesisadaptive.com * extras: mirror.genesisadaptive.com * updates: mirror.genesisadaptive.com Installed Packages plasticscm-certtools-mono4.noarch 3.12.1-1.1 @plasticscm-common plasticscm-client-complete.noarch 5.4.16.918-1.1 @plasticscm-latest plasticscm-client-core.x86_64 5.4.16.918-2.1 @plasticscm-latest plasticscm-client-gtk.noarch 5.4.16.918-1.1 @plasticscm-latest plasticscm-complete.noarch 5.4.16.918-1.1 @plasticscm-latest plasticscm-gnome-sharp-mono4.x86_64 2.24.0-2.1 @plasticscm-common plasticscm-gtk-sharp-mono4.x86_64 2.12.29-2.1 @plasticscm-common plasticscm-mono4.x86_64 4.6.2-2.1 @plasticscm-common plasticscm-server-complete.noarch 5.4.16.918-1.1 @plasticscm-latest plasticscm-server-core.noarch 5.4.16.918-2.1 @plasticscm-latest plasticscm-theme.noarch 5.4.16.918-2.1 @plasticscm-latest Sooo.... I need those log files to stick around, so where do I go from here? Thanks! Neal P.S. I hit submit on this forum post, and it told me I wasn't authorized, and hitting the browser back button also showed me an unauthorized message. Luckily, I was able to recover my message after logging in in a different window and telling the browser to resubmit the POST, but this is dangerous! Link to comment Share on other sites More sharing options...
manu Posted June 2, 2017 Report Share Posted June 2, 2017 Hi! permissions seems to be OK. Can you check if you can run this command? sudo -u plasticscm cp plastic.debug.log.txt plastic.debug.log.txt.roll1 The error is pretty clear though. System.IO.File.Move -> Access to the path is denied I'll try to search related issues with log4net. Link to comment Share on other sites More sharing options...
ngokli Posted June 6, 2017 Author Report Share Posted June 6, 2017 Thanks for the reply, Manu! It seems like most of the log files started rolling properly on Saturday. I don't think I touched anything since my original post, I've been working on other things... I'll restart plasticsd from a terminal before midnight UTC tomorrow to see if there are any error messages (there were on Friday). I just tried running your cp command, and of course it works now! I don't know if it would have worked last week... I'll report back if I find there are still issues. Thanks, Manu! Link to comment Share on other sites More sharing options...
manu Posted June 6, 2017 Report Share Posted June 6, 2017 Ok! No problem. Just let us know if you still notice issues... we can try to get connected together and check if we can see something weird. Happy to know it's working fine Link to comment Share on other sites More sharing options...
ngokli Posted June 7, 2017 Author Report Share Posted June 7, 2017 Aha! I set my ssh shell up to catch the error messages, and they showed up again: log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.relevant.log.txt] -> [plastic.relevant.log.txt.20170606] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20170606] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.channelcall.log.txt] -> [plastic.channelcall.log.txt.20170606] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.errors.log.txt] -> [plastic.errors.log.txt.20170606] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 It seems the log files rolled fine for three days, and then it broke again... ]$ ls -lh *log.txt* -rw-r--r--. 1 plasticscm plasticscm 106K Jun 7 01:33 plastic.channelcall.log.txt -rw-r--r--. 1 plasticscm plasticscm 83K Jun 3 23:59 plastic.channelcall.log.txt.20170603 -rw-r--r--. 1 plasticscm plasticscm 76K Jun 4 23:55 plastic.channelcall.log.txt.20170604 -rw-r--r--. 1 plasticscm plasticscm 162K Jun 5 23:56 plastic.channelcall.log.txt.20170605 -rw-r--r--. 1 plasticscm plasticscm 1.7M Jun 7 01:34 plastic.debug.log.txt -rw-r--r--. 1 plasticscm plasticscm 16M Jun 3 23:59 plastic.debug.log.txt.20170603 -rw-r--r--. 1 plasticscm plasticscm 16M Jun 4 23:59 plastic.debug.log.txt.20170604 -rw-r--r--. 1 plasticscm plasticscm 16M Jun 5 23:59 plastic.debug.log.txt.20170605 -rw-r--r--. 1 plasticscm plasticscm 983 Jun 7 00:14 plastic.errors.log.txt -rw-r--r--. 1 plasticscm plasticscm 558 Jun 3 02:36 plastic.errors.log.txt.20170603 -rw-r--r--. 1 plasticscm plasticscm 558 Jun 4 06:54 plastic.errors.log.txt.20170604 -rw-r--r--. 1 plasticscm plasticscm 558 Jun 5 11:12 plastic.errors.log.txt.20170605 -rw-r--r--. 1 plasticscm plasticscm 2.5M Jun 7 01:33 plastic.realtime.requests.log.txt -rw-r--r--. 1 plasticscm plasticscm 694K Jun 7 01:34 plastic.relevant.log.txt -rw-r--r--. 1 plasticscm plasticscm 11M Jun 3 23:59 plastic.relevant.log.txt.20170603 -rw-r--r--. 1 plasticscm plasticscm 11M Jun 4 23:59 plastic.relevant.log.txt.20170604 -rw-r--r--. 1 plasticscm plasticscm 11M Jun 5 23:59 plastic.relevant.log.txt.20170605 I happened to reboot the server just before this happened... But I've stopped and started plasticsd service itself several times in the past three days. I am able to copy a logfile as plasticscm user. I'm wondering if it's trying to put the rolled logfile in '/' instead of '/opt/plasticscm5/server'. '/' seems to be the working directory of the plasticsd service. On the old server, there are three sets of plastic logfiles from seemingly-random dates in '/'. I haven't found anything special that happened on those three dates... I just modified my loader.log.conf to put my logfiles in '/var/log/plasticscm5' (where logs belong on linux). Maybe specifying an absolute path will help! Thanks, Manu! I'll let you know if I am still having issues... Neal Link to comment Share on other sites More sharing options...
ngokli Posted June 7, 2017 Author Report Share Posted June 7, 2017 By the way, you can ignore 'plastic.realtime.requests.log.txt', because it's set to roll based on size, not date. Link to comment Share on other sites More sharing options...
manu Posted June 8, 2017 Report Share Posted June 8, 2017 On 6/7/2017 at 3:44 AM, ngokli said: I'm wondering if it's trying to put the rolled logfile in '/' instead of '/opt/plasticscm5/server'. '/' seems to be the working directory of the plasticsd service. On the old server, there are three sets of plastic logfiles from seemingly-random dates in '/'. I haven't found anything special that happened on those three dates... That's super weird! But might explain the error... How did you manage to get the plasticd working directory? I'm wondering if log4net is having a log for themselves so we can check the rolling destination path, I'll search for it. It's strange it's working something and sometimes not. Thank you for your time debugging this! To be honest with you this is the first time I face this behavior. Link to comment Share on other sites More sharing options...
Johan Ung Posted November 8, 2017 Report Share Posted November 8, 2017 Hi, did you guys figure this one out? I'm seeing the same issue now. "sudo -u plasticscm" has no problems with permissions but I get these: log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20171107] System.UnauthorizedAccessException: Access to the path is denied. at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0 at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0 This definitely started to happen after a restart (restarted both service and then the whole server machine though). Link to comment Share on other sites More sharing options...
manu Posted November 9, 2017 Report Share Posted November 9, 2017 The thing this is failing from log4net, the library we use for logging... I'm not sure why is this happening or how can I reproduce it I'm wondering if it's something similar to this: https://issues.apache.org/jira/browse/LOG4NET-506 Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.