java.security.AccessControlException: access denied ("oracle.security.jps.service.credstore.CredentialAccessPermission" "context=SYSTEM,mapName=oim,keyName=OIMSchemaPassword" "read")

Issue:

java.security.AccessControlException: access denied ("oracle.security.jps.service.credstore.CredentialAccessPermission" "context=SYSTEM,mapName=oim,keyName=OIMSchemaPassword" "read")

        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:366)

        at java.security.AccessController.checkPermission(AccessController.java:560)

        at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:458)

        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:518)

        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:544)

        Truncated. see log file for complete stacktrace

Caused By: java.security.AccessControlException: access denied ("oracle.security.jps.service.credstore.CredentialAccessPermission" "context=SYSTEM,mapName=oim,keyName=OIMSchemaPassword" "read")

        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:366)

        at java.security.AccessController.checkPermission(AccessController.java:560)

        at oracle.security.jps.util.JpsAuth$AuthorizationMechanism$3.checkPermission(JpsAuth.java:458)

        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:518)

        at oracle.security.jps.util.JpsAuth.checkPermission(JpsAuth.java:544)

        Truncated. see log file for complete stacktrace

Fix/Resolution:

1.       Take the backup of weblogic.policy file to recover easily in case of any issues.

a.        Go to $WLS_HOME/server/lib/weblogic.policy

b.       cp weblogic.policy weblogic.policy_backup

2.       Add the below lines(end of the file) into weblogic.policy file

                        grant codeBase "file:$MW_HOME/patch_wls1036/patch_jars/*" {

permission java.security.AllPermission;

};

3.       Take the backup of system-jazn-data.xml file to recover easily in case of any issues

a.        Go to $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml

b.       $cp system-jazn-data.xml system-jazn-data.xml_backup

4.       Add the below lines (end of the file) into $DOMAIN_HOME/config/fmwconfig/system-jazn-data.xml:

<grant>

  <grantee>

    <codesource>

      <url>file:${wls.home}/../../patch_wls1036/patch_jars/*</url>

    </codesource>

  </grantee>

        <permissions>

                        <permission>
 <class>oracle.security.jps.service.credstore.CredentialAccessPermission</class>

     <name>context=SYSTEM,mapName=oim,keyName=*</name>

     <actions>read,write</actions>

                        </permission>

   </permissions>

</grant>

5.        Restart Admin and managed servers.

            NOTE: Kindly note, MW_HOME will vary from environment to environment depends on your machine path.

                          Also, similar solution is applicable to any component like OAM/OIF/OID etc..

           Hope this post helps you to resolve this issue. 

          

           Thank you for reading out my blog !!

OPSS Unable to start WL Server XML20108 Fatal Error JPS02592 Failed To Push Ldap Config Data To LibOvd

Issue:

 Jun 21, 2016 7:03:16 PM oracle.security.jps.internal.igf.ovd.OvdIGFServiceProvider pushLdapNamesTolibOvd SEVERE: JPS-02592
<Jun 21, 2016 7:03:16 PM SGT> <Error> <Security> <BEA-090892> <The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-02592: Failed to push ldap config data to libOvd for service instance "idstore.ldap" in JPS context "default", cause: org.xml.sax.SAXException: Error Parsing at line #1: 1.org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; <Line 1, Column 1>: XML-20108: (Fatal Error) Start of root element expected.>
<Jun 21, 2016 7:03:16 PM SGT> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-02592: Failed to push ldap config data to libOvd for service instance "idstore.ldap" in JPS context "default", cause: org.xml.sax.SAXException: Error Parsing at line #1: 1.org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; <Line 1, Column 1>: XML-20108: (Fatal Error) Start of root element expected.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-02592: Failed to push ldap config data to libOvd for service instance "idstore.ldap" in JPS context "default", cause: org.xml.sax.SAXException: Error Parsing at line #1: 1.
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; <Line 1, Column 1>: XML-20108: (Fatal Error) Start of root element expected.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-02592: Failed to push ldap config data to libOvd for service instance "idstore.ldap" in JPS context "default", cause: org.xml.sax.SAXException: Error Parsing at line #1: 1.
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; <Line 1, Column 1>: XML-20108: (Fatal Error) Start of root element expected.
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:172)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:375)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        Truncated. See log file for complete stacktrace

Resolution:

1.       Please verify whether your file system having disk space issues ($df -h in case of Linux). One of the causes of XML files corruption is seen when there is a filesystem full condition.

2.       If it is not relates to file system space issues, then it might be because of corrupted adapters.os_xml

3.       As additional symptom $DOMAIN_HOME/config/fmwconfig/ovd/default/server.os_xml was detected as zero (0) bytes.

Corruption of the adapters.os_xml or server.os_xml under this LibOVD location file may lead to Weblogic server startup failure.

Please navigate to the fmwconfig folder and check the adapters.os_xml or server.os_xml file or both
$DOMAIN_HOME/config/fmwconfig/ovd/default/adapters.os_xml

There are chances the any of those xml file could be corrupted due to which the Weblogic server fails to start.

4.       In case the file is corrupted, please perform the following steps:

a.       Take a backup of $DOMAIN_HOME/config/fmwconfig/ovd/default/adapters.os_xml  or  server.os_xml

b.      Delete corrupted adapters.os_xml or server.os_xml files under
 $DOMAIN_HOME/config/fmwconfig/ovd/default/adapters.os_xml

c.       Copy adapters.os_xml or server.os_xml from
         $MW_HOME/oracle_common/modules/oracle.ovd_11.1.1/templates/ to
         $DOMAIN_HOME/config/fmwconfig/ovd/default/. Or restore files from backup.

5.        Restart the server. It should works!!

 Hope this post helps you to resolve this issue.

org.eclipse.persistence.exceptions.DatabaseException Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection

Issue:

****** weblogic startup log ******

INFO: Found persistence provider "org.eclipse.persistence.jpa.PersistenceProvider". OpenJPA will not be used.
[EL Severe]: 2015-12-06

03:35:08.961--ServerSession(1577131615)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.1.v20111018-r10243):

org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLRecoverableException: IO Error: The Network Adapter

could not establish the connection
Error Code: 17002
Dec 6, 2015 3:35:08 AM

oracle.security.jps.internal.credstore.ldap.LdapCredentialStore init
WARNING: Could not create credential store instance. Reason

oracle.security.jps.service.policystore.PolicyStoreConnectivityException: JPS-10000: There was an internal error in the policy store.
JPS-01055: Could not create credential store instance. Reason oracle.security.jps.service.policystore.PolicyStoreConnectivityException:

JPS-10000: There was an internal error in the policy store.
Error: Diagnostics data was not saved to the credential store.
Error:

Validate operation has failed.
Need to do the security configuration first!

Solution:

1. Verify Database listener is up & running or not


2. Take the backup of jpsconfig.xml file


3. Start the database listener


$$ORACLE_HOME/bin

$lsnrctrl status listener  -- Verify Listener status

$lsnrctrl start listener   -- Start the listener,if already not started


4. Start the weblogic from command prompt


$DOMAIN_HOME/bin

$nohup ./startWeblogic.sh > AdminServer.log &

$tail -f AdminServer.log

5. Weblogic Admin Server will start normally.

How to checks the logs for Weblogic/SOA/OIM and OAM Servers in Linux

Server Log locations :

1. Weblogic Server : $DOMAIN_HOME/servers/Admin_server and view admin_server_diagnostic.log

2. SOA Server : $DOMAIN_HOME/servers/soa_server1 and view soa_server1_diagnostic.log

3. OIM Server : $DOMAIN_HOME/servers/oim_server1 and view oim_server1_diagnostic.log

4. OAM Server : $DOMAIN_HOME/servers/oam_server1 and view oam_server1_diagnostic.log

Main difference between Application Server and Web Server

1. Application Server supports distributed transaction and EJB. While Web Server only supports Servlets and JSP.

2. Application Server can contain web server in them. most of App server e.g. JBoss or WAS has Servlet and JSP container.

3. Though its not limited to Application Server but they used to provide services like Connection pooling, Transaction management, messaging, clustering, load balancing and persistence. Now Apache tomcat also provides connection pooling.

4. In terms of logical difference between web server and application server. web server is supposed to provide http protocol level service while application server provides support to web service and expose business level service e.g. EJB.

5. Application server are more heavy than web server in terms of resource utilization.

Weblogic Password Decrypt script

Copy the below python script as PasswordDecryption.py

import os
import weblogic.security.internal.SerializedSystemIni
import weblogic.security.internal.encryption.ClearOrEncryptedService
def decrypt(domainHomeName, encryptedPwd):
domainHomeAbsolutePath = os.path.abspath(domainHomeName)
encryptionService = weblogic.security.internal.SerializedSystemIni.getEncryptionService(domainHomeName)
ces = weblogic.security.internal.encryption.ClearOrEncryptedService(encryptionService)
clear = ces.decrypt(encryptedPwd)
print "RESULT:" + clear
try:
if len(sys.argv) == 3:
decrypt(sys.argv[1], sys.argv[2])
else:
print "INVALID ARGUMENTS"
print " Usage: java weblogic.WLST decryptPassword.py DOMAIN_HOME ENCRYPTED_PASSWORD"
print " Example:"
print " java weblogic.WLST decryptPassword.py D:/Oracle/Middleware/user_projects/domains/base_domain {AES}819R5h3JUS9fAcPmF58p9Wb3syTJxFl0t8NInD/ykkE="
except:
print "Unexpected error: ", sys.exc_info()[0]
dumpStack()
raise

Usage:

./wlst.sh <PATH_OF ATTACHED FILE>  <WLDOMAIN_HOME> <EncryptedPassword>

e.g.

./wlst.sh /tmp/PasswordDecryptor.py <MIDDLEWARE_HOME>/user_projects/domains/<IDM DOMAIN>  {AES}YJKUITRFGTYUHH45YHHGGYFV879655HYU\=

Lost Weblogic Password Reset

Steps to Reset WebLogic password:

1.       Set MW_HOME and DOMAIN_HOME

2.       Go to $DOMAIN_HOME/bin -> ./setDoaminEnv.sh (linux) // setDomainEnv.cmd (Windows)

3.       mv $DOMAIN_HOME/servers/AdminServer/data $DOMAIN_HOME/servers/AdminServer/data-old

a.       <Middleware_Home>\user_projects\domains\base_domain\servers\AdminServer\data\ldap as it will get recreated once the WebLogic Admin server starts.

4.       set CLASSPATH as well. Ex: export CLASSPATH=$WLS_HOME/server/lib/weblogic.jar

5.       Go to $DOAMIN/security

6.       mv DefaultAuthenticatorInit.ldift DefaultAuthenticatorInit_old.ldift

7.       $java weblogic.security.utils.AdminAccount <weblogic> <Newpassword> . (If java path is not set, then try to provide java full path location like /app/java/jdk1.6/bin/java etc…)

8.       $DOMAIN_HOME/servers/AdminServer/security/boot.properties (If you want to set the boot.properties)

a.       Username=weblogic

b.      Password=Newpassword

9.       $DOMAIN_HOME/bin -> ./startWebLogic.sh (Linux) // startWebLogic.cmd (Windows)

Steps to Enable display on Linux machine:

Steps:

1. Switch to root user (If not in already):

            $ su -
           Enter password

2. Type the below command as root user:

          # export DISPLAY=machine:0.0;

3. Check it is set or not by entering the below command:

         #echo $DISPLAY

4. Disable the hostname access

         # xhost + (or) # xhost + hostname

5. Check GUI is working or not by entering the below command:

       # xclock

6. Switch to normal user :

      # su - username
     Enter password

7. Type below commands to access the GUI:

     $ xhost  +

8. Type xclock command in the command prompt

Note: If we need the same GUI in windows machine, need to install X client or windows related display tools.

A timeout occurred while interacting with oim_server1

A timeout occurred  while interacting with oim_server1. Limited Information Available

Solution: 

Just  restart the server by using weblogic console or Kill the server process from the command prompt 

1. How to know the process id in Linux:

            ps  -ef | grep pid

            Ex: ps  -ef | grep java

2. How to kill the process id in Linux

            ps  -ef | grep pic

            Ex: ps  -ef | grep 1121

java.lang.OutOfMemoryError: Java heap space

 ####<Jun 25, 2014 7:17:30 PM SGT> <Error> <Cluster> <lapoid02> <oim_server1> <weblogic.cluster.MessageReceiver> <<WLS Kernel>> <> <a1a043141f07f93c:63d55d3a:1468e87738f:-8000-0000000000011252> <1403695050414> <BEA-003108> <Unicast receive error : java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

Solution:  Just increase the Heap Size with the below guide lines through Weblogic Console

1.       Login to Weblogic Console -> Click on Lock and Edit button -> Go to Environment -> servers -> oim_server1 -> configuration -> server Start and change/update the parameters in "Arguments" field -> Enter the values:   -XX:MaxPermSize=4096m -Xms4096m -Xmx4096m (provide your own values)

2.       Save and Release the Configuration

3.       Restart oim_server1 

If you are still facing the issue, and unable to stop the servers through weblogic console, then you can directly kill the pid’s using the command kill -9 pid 

Application Server and Web server

Application Server and Web server:

• A Web server exclusively handles HTTP requests, whereas an application server serves business logic to application programs through any number of protocols.
• Webserver mainly handles the Http requests but app server can be used to handle the http, rmi, TCP/IP and many more protocols.  In case of application server, it does the same thing, of getting and gives the response but it can process the requests. The web server can be considered as the subset of app server

Difference between a web server and an application server:

• WebServer can execute only web applications i.e. servlets and JSPs and has only a single container known as Web container which is used to interpret/execute web applications
• Application server can execute Enterprise application, i,e (servlets, jsps, and EJBs) it is having two containers 
• WebContainer (for interpreting/executing servlets and jsps)
• EJB container (for executing EJBs). It can perform operations like load balancing , transaction demarcation etc

Configure One-way SSL

Configure One-way SSL

By default, SSL is enabled and configured to use the demonstration Identity and Trust keystores. For testing and development purposes, the SSL configuration is complete.

Use the steps in this section to configure SSL for production use.

To configure SSL:

• Expand the Servers node.
• Select the name of the server for which you want to configure keystores (for example, exampleserver).
• Select the Configuration-->Keystores and SSL tab.
• Information about the demonstration Identity and Trust keystores is displayed in the Keystore Configuration.
• Configure new Identity and Trust keystores for WebLogic Server. 
• Click the Change... link in the SSL Configuration to configure attributes for SSL.
• The Configure SSL page appears.
• Specify how the identity and trust for WebLogic Server is stored.
•  The following options are available:
• Key Stores—Use this option if you created Identity and Trust keystores for WebLogic Server. If you choose this option, go to step 8.
• Files or Key Store Providers—Use this option if you stored private keys and trusted CA certificates in a file or in a JKS keystore accessed via the WebLogic Keystore provider (as supported in previous releases of WebLogic Server). If you choose this option, go to step 9. This option is available for the purpose of backward compatibility only and it automatcally set with security information from a previous release of WebLogic Server.
• Click Continue.
• Specify the alias used to load the private key into the keystore in the Private Key Alias and the password used to retrieve the private key from the keystore in the Passphrase attibute. You may have specified this information when creating the Identity keystore; however, for the purpose of SSL configuration specify the information again. 
• Note: You do not have to specify this information for the Trust keystore because trusted CA certificates are not individually identified to WebLogic Server with aliases. All trusted CA certificates in a keystore identified as trusted by WebLogic Server are trusted. Therefore, WebLogic Server does not require an alias when retrieving a trusted CA certificate from the keystore.
• Specify information about the location of identity and trust for WebLogic Server.
• Note: This step only applies if the Files or Key Store Providers option is specified.
• Private Key File Name—The directory location of the private key for WebLogic Server. Specify a value for this attribute only if you stored the private key for WebLogic Server in a file (versus a WebLogic Keystore provider).
• Private Key Alias—The alias specified when loading the private key for WebLogic Server from the keystore. Specify a value for this field only if you stored the private key for WebLogic Server in a keystore accessed by the WebLogic Keystore provider.
• Passphrase—The password specified when loading the private key for WebLogic Server into the keystore. Specify a value for this field only if you stored the private key for WebLogic Server in a keystore accesssed by the WebLogic Keystore provider. Confirm the password. If you protected the private key file with a password, specify the weblogic.management.pkpassword command-line argument when starting the server.
• Server Certificate File Name— The directory location of the digital certificate for WebLogic Server. If you are using a certificate chain that is deeper than two certificates, you to need to include the entire chain in PEM format in the certificate file.
• Trusted CA File Name—The name of the file containing the PEM-encoded trusted certificate authorities.
• Click Continue.
• Click Finish.
• Reboot WebLogic Server

Configuring Two-Way SSL

Configuring Two-Way SSL

By default, WebLogic Server is configured to use one-way SSL (the server passes its identity to the client). 

For a more secure SSL connection, use two-way SSL. In a two-way SSL connection, the client verifies the identity and trust of the server and then passes its identity to the server. The server then validates the identity and trust of the client before completing the SSL connection. The server determines whether or not two-way SSL is used.

Before configuring two-way SSL, ensure the Trust key store for the server includes the certificate for the trusted certificate authority that signed the certificate for the client.

To enable two-way SSL:

First Configure one-way SSL then follow the below steps for two-way SSL

• Expand the Servers node.
• Select the name of the server for which you want to configure two-way SSL (for example, exampleserver).
• Select the Configuration-->Keystores and SSL tab.
• Click the Show link under Advanced Options.
• Go to the Server attributes section of the window.
• Set the Two Way Client Cert Behavior attribute. 
• The following options are available:
• Client Certs Not Requested—The default (meaning one-way SSL).
• Client Certs Requested But Not Enforced—Requires a client to present a certificate. If a certificate is not presented, the SSL connection continues.
• Client Certs Requested And Enforced—Requires a client to present a certificate. If a certificate is not presented or if the certificate is not trusted, the SSL connection is terminated.
• Click Apply.

Reboot WebLogic Server.

SSL EXCEPTIONS IN ADMIN SERVER AND NODE MANAGER

SSL EXCEPTIONS IN ADMIN SERVER AND NODE MANAGER

Exception: 

javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from oracle.test.com – xx.xxx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected.

Solution:

The above exception is one of the most common exception encountered during the setup of Weblogic Server in an environment. The stack does suggest what could be the reasons but the diagnostics are not mentioned.

To debug this issue, first we need to check the certificates used by Admin Server and the Node Manager. If we have Admin and the Node Manager using demo certificates, then the issue can be due to improper DNS mapping. 

We can use the nslookup to check the DNS entry. For testing purpose we can provide the ip address as the listen address for the admin server and the node manager and see if the issue is still occurring.

Also we will have to turn off host name verification and the basic validation check of the certificates. 
We can do it by specifying the following flag in startWeblogic.sh

-Dssl.debug=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.enforceConstraints=off

And the following flag in startNodeManager.sh

-Dssl.debug=true -Dweblogic.nodemanager.sslHostNameVerificationEnabled=false -Dweblogic.security.SSL.enforceConstraints=off

If the Admin Server is using Custom Identity and Custom trust, then its better to configure the node manger with custom identity and custom trust as well. 

By default the Node Manager is configured with Demo Identity and Demo Trust. To change it to custom identity and custom trust, we need to specify the following values in the nodemanager.properties file present in nodemanager home

Keystores=CustomIdentityandCustomTrust
CustomIdentityAlias= 
CustomIdentityKeyStoreFileName= 
CustomIdentityKeyStorePassPhrase = xxxxxx
CustomIdentityKeyStoreType = JKS
CustomIdentityPrivateKeyPassPhrase = xxxxxxx

Apply the same flags as above in the startup script of Admin Server and Node Manger.

Check from the console whether Node Manager is reachable or not.

Another option can be to use PLAIN communication between Admin Server and Node Manager.

We can change the Listen Type to PLAIN for the Node Manager from the console and set the secureListener=false in the nodemanager.properties file present in nodemanager home.

Please find the below URL for your reference:

http://download.oracle.com/docs/cd/E15051_01/wls/docs103/nodemgr/nodemgr_config.html#wp1101097